Enforce MAX BYTES for transferring batch records

infinyon / fluvio

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.

https://www.fluvio.io/

Apache License 2.0

3.88k stars 491 forks source link

Closed sehz closed 4 years ago

sehz commented 4 years ago

Fluvio uses zero copy I/o to transfer records from SPU to client. Records to need to be limit since maximum bytes can be up to 1G for each segment.

Solution:

SPU: Limit transferring file slice. This will create potentially truncated last batch
Kf decoder: When decoding DefaultRecords, ignore any batch that can't be decoded
Client: Use offset in the last batch to retrieve next batches for streaming cases
SPU-SPU: Apply spu-message-size to limit bytes

sehz commented 4 years ago

Everything except item 3 is implemented. That will be done as part of Issue #72 (new client API)