search

zendesk / maxwell

Maxwell's daemon, a mysql-to-json kafka producer
https://maxwells-daemon.io/
Other
4.02k stars 1.01k forks source link

Inconsistent timestamp resolutions #1306

Open acrois opened 5 years ago

acrois commented 5 years ago

I've checked out related issues and it seems like nothing has been done to address normalizing the timestamps returned by DDL statements.

Currently: DDL events have millisecond precision CDC events have second precision

Would be great to have a flag to standardize this to milliseconds or seconds. I've done the normalization in the application processing the maxwell events, but it should really be solved up stream.

Thoughts? I'd be happy to work on this issue and submit PR when it comes time, just want to make sure this is worthwhile and hasn't been addressed in some other way.

osheroff commented 5 years ago

unfortunately CDC events will never have better than second precision, due to some fairly premature optimization in the mysql binlog format. I'm not utterly sure why DDL events do have millisecond precision, a PR to set that back to seconds would be appreciated.

If you take this on, you should probably introduce a new field in DDL events instead of just suddenly changing the old field. Or a flag to setup behavior.