Support for compacted event types

[ePaul]

Background / Status quo

Currently this library does not support submitting events to compacted event types, as it is missing the partition_compaction_key in the metadata object.

This is justified by these paragraphs in the README:

Be aware that this library does neither guarantee that events are sent exactly once, nor that they are sent in the order they have been persisted. This is not a bug but a design decision that allows us to skip and retry sending events later in case of temporary failures. So make sure that your events are designed to be processed out of order (See Rule 203 in Zalando's API guidelines). To help you in this matter, the library generates a strictly monotonically increasing event id (field metadata/eid in Nakadi's event object) that can be used to reconstruct the message order.

Unfortunately this approach is not compatible with Nakadi's compacted event types – it can happen that the last event submitted (and thus the one which will stay after compaction) is not the last event which was actually been fired. For this reason, the library currently also doesn't provide any access to Nakadi's partition_compaction_key feature.

There are some cases where the out-of-order submission is less critical – e.g. when you have occasionally have groups of events for different entities (and thus different compaction keys), and a long time in-between those groups. Here is it quite unlikely that events for the same entity are submitted out of order, and compaction is also quite useful.

Possible workarounds for this situation:

• don't use transactional event production (and use a Nakadi client directly)
• reimplement the transactional event production, ignore the out-of-order problem (or set up some monitoring to fix it manually when it happens, e.g. then sending snapshot events)
• reimplement transactional event production in a way that stops sending out events when one can't be submitted (so the order is kept)
  • (this is what Fabric Event Streams [a Zalando-internal solution for event-streaming from the DB's WAL] is doing by default.)
• reimplement transactional event production in a way that stops sending out events for the same compaction key when one can't be submitted (so the order per compaction key is kept)

Suggestion

1. Make it possible to provide the compaction key when producing events with this library (of course optional, and in a compatible way so old code doesn't break)
2. Still keep a warning similar to the one above in the README (and in the javadoc of this feature), so people are not surprised when they still get the not-last event in their compacted event type.

[ePaul]

Implementation plan

For the compatible API, I see these options here:

• add copies of all the fire* methods to EventLogWriter, with an additional compactionKey parameter.
• Give the application an option to register a CompactionKeyExtractor (basically a BiFunction<String, Object, String>), which is then called internally when any of the fire* methods are called.
  • This is somewhat complicated, but would leave the actual event production API alone. That could benefit applications which build some internal framework around this, where the event-producing code doesn't know whether the event type is compacted (or what the key is supposed to be), but some other part of the same application knows it. (Sounds a bit contrived, though.)
• We could also have one extractor (then just a Function<Object, String>) per event type.

In addition, we need this:

• add the field to EventLog class
• have the fire-methods fill the field when creating the EventLog object
• add flyway migration to extend the DB table
• extend the repository methods to store/fetch this field
• add it to NakadiMetadata
• add it to the mapper in EventTransmissionService

[ePaul]

So I built two variants of this:

• In #158, via a registerCompactionKeyExtractor method on the EventLogWriter (→ release candidate 21.0.0-RC3a), and
• in #164, via registering spring beans of type CompactionKeyExtractor (→ release candidate 21.0.0-RC3b).

We tried this out in an internal service, and it the second way looks cleaner, so I'd suggest to go for this way.

[ePaul]

164 was merged already half a year ago (and released in 21.0.0), so I guess we can close this now as completed.