search

dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.65k stars 943 forks source link

Question: Migrate from Redis Cloud (using a RDB File) to DragonFly, but excluding Timeseries #3971

Open wernermorgenstern opened 1 day ago

wernermorgenstern commented 1 day ago

So, we are currently using Redis Cloud, and we have hourly snapshots (RDB Files) to S3. We want to migrate our data to DragonFly.

However, we are using TimeSeries in Redis Cloud, and DragonFly does not support Timeseries. When I tried to use the RDB File, per the Migration Documentation (https://www.dragonflydb.io/docs/migrating-to-dragonfly/from-redis-instance/snapshot-and-restore), after starting up DragonFly, I got an error Unknown Type: TSDB (something like that), which makes sense.

Does anybody have any ideas how to filter out the Timeseries Data out of the RDB File? I have tried two tools (RDB Tools), but both of them didn't work for me (one of them did convert to a JSON file, but it doesn't have an option to convert back to a RDB File.

There might be a few options in my mind, but all of them require some downtime. One option:

  1. In Redis Cloud, create a new database
  2. Load the RDB File into that Database.
  3. Remove the Keys, which are Timeseries
  4. Create a Backup (RDB)
  5. Try to use that file for DragonFly

This is not tested though.

Thank you

wernermorgenstern commented 1 day ago

So, an update after testing this on a Test Database:

  1. In Redis Cloud, create a new database (with RediSearch and Timeseries module enabled)
  2. Load the RDB File into that Database.
  3. Remove the Keys, which are Timeseries
  4. Create a Backup (RDB)
  5. In Redis Cloud, create another new database (with RediSearch enabled) - No Timeseries
  6. Load the RDB from Step 4 into that database
  7. Create a Backup
  8. Use that RDB file for Dragonfile.

When I start DragonFly, it works. I do see though a few less keys (like around 100)

And I see these warnings:

dragonfly-agent-all  | I20241023 14:31:26.708123     1 server_family.cc:1084] Loading /data/dump.rdb
dragonfly-agent-all  | W20241023 14:31:26.709416    12 rdb_load.cc:2600] Unrecognized RDB AUX field: 'aof-base'
dragonfly-agent-all  | W20241023 14:31:26.709452    12 rdb_load.cc:2289] WARNING: Skipping data for module scdtype00
dragonfly-agent-all  | W20241023 14:31:26.709456    12 rdb_load.cc:2289] WARNING: Skipping data for module ft_index0
dragonfly-agent-all  | I20241023 14:31:26.715684    12 listener_interface.cc:101] sock[9] AcceptServer - listening on port 6390
dragonfly-agent-all  | W20241023 14:31:26.780007    12 rdb_load.cc:2289] WARNING: Skipping data for module scdtype00
dragonfly-agent-all  | I20241023 14:31:26.780550    11 server_family.cc:1137] Load finished, num keys read: 16904

it means some downtime, when we migrate from Redis Cloud to DragonFly.