Retry deleting messages without resending the batch if the delete fails (fixes #34)
Delete messages in batches rather than all at once to cut down on the chance of a deadlock.
Add last_processed_at to kafka_topic_info to ensure that wait metrics are accurate in cases where records get created with an old created_at time (e.g. for long-running transactions).
Type of change
Please delete options that are not relevant.
[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
How Has This Been Tested?
Tested in production with relatively heavy load and an old MySQL database.
Checklist:
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
last_processed_attokafka_topic_infoto ensure that wait metrics are accurate in cases where records get created with an oldcreated_attime (e.g. for long-running transactions).Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Tested in production with relatively heavy load and an old MySQL database.
Checklist: