linkedin / venice

Venice, Derived Data Platform for Planet-Scale Workloads.
https://venicedb.org
BSD 2-Clause "Simplified" License
487 stars 84 forks source link

[compat][server][client][test] Global RT DIV improvement (part 2): Ch… #1257

Open lluwm opened 1 week ago

lluwm commented 1 week ago

…unking support for DIV message

This change mainly focuses on adding chunking support for DIV messages when they are produced to Kafka topics, as the size of DIV message can be large. We leverage today's chunking mechanism for regular records and extend it to support DIV with the following modifications:

  1. All the DIV messages are of type CONTROL_MESSAGE_DIV in its KafkaKey and their corresponding KafkaMessageEnvelope uses Put as the payload.
  2. Inside the Put payload, the actual message is stored in the putValue field and the schemaId can have the following cases:
    • If the DIV message is non-chunked, the schemaId is set to GLOBAL_DIV_STATE.
    • If the DIV message is chunk message, the schemaId is set to CHUNK.
    • If the DIV message is a chunk manifest message, the schemaId is set to CHUNKED_VALUE_MANIFEST.
  3. ChunkAssembler is adapted, on the receiver side, to buffer, assemble, and deserialize DIV messages (chunked/non-chunked).

How was this PR tested?

Does this PR introduce any user-facing changes?