markpapadakis
commented
5 years ago We could also use another mode where we retain all messages (i.e not the first or the last message for a given key), which could be handy if you only want to merge multiple immutable segments into fewer.
@kostas on @dist-sys(Slack) came up with a great idea. Instead of retaining the last message in a set of messages with the same key, retain the first instead (during compactions). That way, you can implement an indempotency scheme, where you really care for the first message, and if you get more you will ignore them anyway, and you want them to be purged, eventually.