feat: add an option to flush serialized entries on threshold limit

kostasrim commented 3 months ago

Resolves #3223

serialize big slots in chunks
allow preemption on large slots
disable big entries serialization for RDB files
add test

There is also a separate issue with compression. Previously, we only called FlushToSink in SaveEpilogue and before SaveBody (to flush previous entries). However, with the current changes, we call FlushToSink when we serialize X number of bytes. This doesn't work with the protocol implemented in rdb_load because we now split let's say a list in multiple chunks and LoadKeyValue on RdbLoader expects to read a string (for the values) but it gets a compressed blob and fails with unrecognized rdb type. I created this separate issue for that: #3324

romange commented 2 months ago

what is the related issue for this PR?

kostasrim commented 2 months ago

what is the related issue for this PR?

I will edit the description soon, it's still in draft and incomplete. Plz don't review

dragonflydb / dragonfly

feat: add an option to flush serialized entries on threshold limit #3241