Open pbardea opened 3 years ago
cc @cockroachdb/cdc
I think we still want to do this. At the very least, we need memory monitoring around the buffer in the ingestion processor.
Put some time on reusing the buffer_adder in stream_ingestion_processor, it can end up very messy code. After some research on this, I think we better have its own buffer_adder, sst batcher and stats for stream ingestion:
However, we can still reuse the kvBuf for the way it encode keys and values, it needs some modifications mostly how we sort it kvBufEntry if we want to encode the key and timestamp into one fat key.
Currently the stream ingestion processor uses a slice as it's buffer before it flushes. We should have some sort of
BufferingAdder
that supports inserting keys at a particular timestamp. Concrete actions here:kvBuf
type buffer that supports storing MVCC KVs.Jira issue: CRDB-3325
Epic CRDB-19048