search

dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.96k stars 954 forks source link

chore: Reduce memory consumption when migrating huge values #4119

Closed chakaz closed 5 days ago

chakaz commented 1 week ago

Before this PR:

We serialized a RESTORE command for each entry into a string, and then push that string to the wire. This means that, when serializing an entry of size X, we consume 2X memory during the migration.

This PR:

Instead of serializing into a string, we serialize into the wire directly. Luckily, only a small modification is needed in the way we interact with crc64, which works well even in chunks.

Fixes #4100

kostasrim commented 1 week ago

Anddd welcome back :partying_face:

BorysTheDev commented 1 week ago

@chakaz I think we need the same for journalStreamer/RestoreStreamer