As of Decaton 8.0.1, the most common way to use Decaton was to use DecatonClient as the producer and DecatonProcessor as the consumer, with implementing value Serializer/Deserializer respectively.
After #238, DecatonClient no longer wraps tasks with DecatonTaskRequest protobuf, which means records produced by DecatonClient and ones produced by non-DecatonClient producers (serialized by Kafka's serializer) have identical serialized bytes.
Then why not making Deserializer support consuming non-DecatonClient topics, without forcing users to implement TaskExtractor (which is bit bothersome than just a deserializer) ?
Summary of changes
Update documentations and examples
ProcessorsBuilder#consuming(String topic, Deserializer<T> deserializer) now deserializes messages bytes directly even when task-metadata header is missing. This is a breaking change
Previous behavior when task-metadata header is missing was to unwrap as DecatonTaskRequest pb first, then deserialize serialized task in it. This was for <= 8.0.0 to 9.0.0 migration where old DecatonClient might produce records during the upgrade.
Since "records are produced by old client / or records are produced by non-Decaton client" is not distinguishable from Decaton, we add new decaton.parse.as.legacy.format.when.header.missing property to control the behavior for graceful migration.
With regards to this, when task-metadata header is missing, we need to fill reasonable default of task metadata.
The only field must be filled might be TaskMetadata#timestampMillis.
e.g. to calculate delivery latency correctly
To fill timestampMillis, we add ConsumedRecord#recordTimestampMillis, which can be retrieved from record itself
Expected upgrade procedure from <= 8.0.1 to 9.0.0
Enable both decaton.retry.task.as.legacy.format and decaton.parse.as.legacy.format.when.header.missing
Motivation
DecatonTaskRequestprotobuf, which means records produced by DecatonClient and ones produced by non-DecatonClient producers (serialized by Kafka's serializer) have identical serialized bytes.Deserializersupport consuming non-DecatonClient topics, without forcing users to implementTaskExtractor(which is bit bothersome than just a deserializer) ?Summary of changes
ProcessorsBuilder#consuming(String topic, Deserializer<T> deserializer)now deserializes messages bytes directly even when task-metadata header is missing. This is a breaking changeDecatonTaskRequestpb first, then deserialize serialized task in it. This was for<= 8.0.0 to 9.0.0migration where old DecatonClient might produce records during the upgrade.decaton.parse.as.legacy.format.when.header.missingproperty to control the behavior for graceful migration.TaskMetadata#timestampMillis.timestampMillis, we addConsumedRecord#recordTimestampMillis, which can be retrieved from record itselfExpected upgrade procedure from <= 8.0.1 to 9.0.0
decaton.retry.task.as.legacy.formatanddecaton.parse.as.legacy.format.when.header.missingdecaton.retry.task.as.legacy.formatdecaton.parse.as.legacy.format.when.header.missingI'll describe the detail in 9.0.0 release note