Originally this PR intended to enable topic compaction on the workflow/task topics. However after much research it appears that for our application the current implementation of topic compaction could result in inconsistent information.
Before enabling compaction we need to ensure that we are passing our event UTC time to some yet to be developed custom header for compaction to use to determine order otherwise order is not guaranteed.
Originally this PR intended to enable topic compaction on the workflow/task topics. However after much research it appears that for our application the current implementation of topic compaction could result in inconsistent information.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction
Before enabling compaction we need to ensure that we are passing our event UTC time to some yet to be developed custom header for compaction to use to determine order otherwise order is not guaranteed.