search

JeffersonLab / jaws

An alarm system built on Kafka that supports pluggable sources
https://ace.jlab.org/jaws
MIT License
7 stars 0 forks source link

Test and Review Timestamp Behavior #30

Open slominskir opened 2 years ago

slominskir commented 2 years ago

We need to do some testing to ensure timestamps found in Kafka messages make sense. I believe we're using producer provided timestamps (EPICS IOCs), which we may want to re-think. Broker (Kafka server) is the other option. We could include both if we add one to the message payload. Not sure what should happen if they're very different. Sounds like an IOC that's drifted time-wise could cause havoc on Kafka Stream joins.

There has been questions about providing timestamps for both actual (direct-from-producer) and computed (effective) messages, and at this point I believe they're identical but testing is needed. The docs here suggest that a Kafka Stream join of a KTable result in max(left.ts, right.ts).

https://docs.confluent.io/platform/current/streams/concepts.html#time

slominskir commented 1 month ago

See Also: https://github.com/JeffersonLab/epics2kafka/issues/1

I think I came to the wrong conclusion on that issue. The safest option is likely to override the default and configure global LogAppendTime.

See: https://github.com/JeffersonLab/jaws/commit/d0de43b33b621098c9475fdefd04cd1525b86c58