This change mainly focuses on adding chunking support for DIV messages when they are produced to Kafka topics, as the size of DIV message can be large. We leverage today's chunking mechanism for regular records and extend it to support DIV with the following modifications:
All the DIV messages are of type CONTROL_MESSAGE_DIV in its KafkaKey and their corresponding KafkaMessageEnvelope uses Put as the payload.
Inside the Put payload, the actual message is stored in the putValue field and the schemaId can have the following cases:
If the DIV message is non-chunked, the schemaId is set to GLOBAL_DIV_STATE.
If the DIV message is chunk message, the schemaId is set to CHUNK.
If the DIV message is a chunk manifest message, the schemaId is set to CHUNKED_VALUE_MANIFEST.
ChunkAssembler is adapted, on the receiver side, to buffer, assemble, and deserialize DIV messages (chunked/non-chunked).
How was this PR tested?
new integration test.
passed CI
Does this PR introduce any user-facing changes?
[x] No. You can skip the rest of this section.
[ ] Yes. Make sure to explain your proposed changes and call out the behavior change.
…unking support for DIV message
This change mainly focuses on adding chunking support for DIV messages when they are produced to Kafka topics, as the size of DIV message can be large. We leverage today's chunking mechanism for regular records and extend it to support DIV with the following modifications:
How was this PR tested?
Does this PR introduce any user-facing changes?