search

dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.96k stars 954 forks source link

feat: Huge values breakdown in cluster migration #4144

Open chakaz opened 3 days ago

chakaz commented 3 days ago

Before this PR we used RESTORE commands for transferring data between source and target nodes in cluster slots migration.

While this works, it has a side effect of consuming 2x memory for huge values (i.e. if a single key's value takes 10gb, serializing it will take 20gb or even 30gb).

With this PR we break down huge keys into multiple commands (RPUSH, HSET, etc), respecting the existing --serialization_max_chunk_size flag.

Note that we still use RESTORE serialization for small values and for some data types which are not (yet?) supported.

Part of #4100

chakaz commented 22 hours ago

@adiholden following our offline sync, I modified this PR (including desc) to only do sync breakdown for huge values. I'll follow up on async later.