Importing the data ERROR

[linydquantil]

redis version: 5.0.4 redis-dump: 0.4.6

[root@mtl-gslb-redis ~]# redis-cli info
# Keyspace
db0:keys=6890,expires=0,avg_ttl=0
db1:keys=86949,expires=0,avg_ttl=0             large keys is in db1
db2:keys=1217,expires=0,avg_ttl=0
db3:keys=8196,expires=0,avg_ttl=0
db4:keys=493653,expires=0,avg_ttl=0
db5:keys=1178,expires=0,avg_ttl=0
db6:keys=1,expires=0,avg_ttl=0
db8:keys=1,expires=0,avg_ttl=0
db12:keys=155,expires=0,avg_ttl=0

redis dump success but import data error:

[root@mtl-gslb-redis ~]# redis-cli -n 2  --pipe < 2.resp    
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1218
[root@mtl-gslb-redis ~]# redis-cli -n 1 --pipe < 1.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
ERR Protocol error: invalid multibulk length
Error reading from the server: Connection reset by peer

[yannh]

Hi @linydquantil , I m assuming 1.resp is db1 and 2.resp is db2?

[yannh]

Could you actually give a shot at reporting against redis-cli? It looks like it might have its own timeouts... a file with garbage content that would allow to reproduce this would be super useful.
Also - could you time the failing import, and give us the size of 1.resp? How big is the single key, 200MB?

[yannh]

One simpler thing we could try is set the Zset in multiple small commands rather than a single large one. Still this is unusual :(

[linydquantil]

right, 1.resp is db1 and 2.resp is db2

[root@mtl-gslb-redis ~]# ls -lh
total 901M
-rw-r--r-- 1 root root 220K Apr 23 09:07 0.resp
-rw-r--r-- 1 root root 605M Apr 23 09:08 1.resp
-rw-r--r-- 1 root root 894K Apr 23 09:09 2.resp

The size of this key is larger than that of 200MB.

[linydquantil]

@mpartio do you have any idea?

[mpartio]

I tracked down a similar problem but I didn't find the root cause. I think it is somehow related to redis itself, not redis-dump-go. When the file-to-be-imported was sufficiently large, the import just crashed.

[yannh]

I will consider writing an improvement to import large zsets/maps in multiple commands :bow:

[yannh]

Alright I'm able to reproduce https://github.com/yannh/redis-dump-go/compare/integration-tests?expand=1 Using my mock zset I m starting to see issues once the zset has around 1M entries.

-> Filling Redis with Mock Data...
OK
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1000000
-> Dumping DB...
Database 0: 1 element dumped
-> Flushing DB and restoring dump...
OK
Error writing to the server: Connection reset by peer``` 

[yannh]

Released v0.5.0 - long commands are now broken down into batches to avoid errors during imports. Also: there is an acceptance tests that generates, imports, dumps a ZSET with 1M entries, and then reimports it - to ensure this does not break in the future.

[linydquantil]

this is my test result, the tool version: 0.5.0

1. get redis info

   
   redis-cli info

Keyspace

db0:keys=6891,expires=1,avg_ttl=48157 db1:keys=88555,expires=0,avg_ttl=0 large keys is in db1 db2:keys=1165,expires=0,avg_ttl=0 db3:keys=8410,expires=0,avg_ttl=0 db4:keys=456667,expires=0,avg_ttl=0 db5:keys=1182,expires=0,avg_ttl=0 db6:keys=6422,expires=0,avg_ttl=0 db8:keys=1,expires=0,avg_ttl=0 db12:keys=155,expires=0,avg_ttl=0

2. dump data

[root@mtl-gslb-redis ~]# ./redis-dump-go -db 0 > 0.resp Database 0: 6891 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 1 > 1.resp
Database 1: 88555 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 2 > 2.resp
Database 2: 1165 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 3 > 3.resp
Database 3: 8410 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 4 > 4.resp
Database 4: 456667 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 5 > 5.resp
Database 5: 1182 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 6 > 6.resp
Database 6: 6422 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 8 > 8.resp
Database 8: 1 element dumped [root@mtl-gslb-redis ~]# ./redis-dump-go -db 12 > 12.resp
Database 12: 155 element dumped

the resp file list is -rw-r--r-- 1 root root 871K Apr 30 01:11 0.resp -rw-r--r-- 1 root root 56K Apr 30 01:18 12.resp -rw-r--r-- 1 root root 491M Apr 30 01:12 1.resp -rw-r--r-- 1 root root 754K Apr 30 01:14 2.resp -rw-r--r-- 1 root root 14M Apr 30 01:14 3.resp -rw-r--r-- 1 root root 33M Apr 30 01:17 4.resp -rw-r--r-- 1 root root 1.1M Apr 30 01:18 5.resp -rw-r--r-- 1 root root 645K Apr 30 01:18 6.resp -rw-r--r-- 1 root root 796 Apr 30 01:18 8.resp

3. flushall redis

redis-cli flushall

4. import data

[root@mtl-gslb-redis ~]# redis-cli -n 0 --pipe < 0.resp Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 6893 [root@mtl-gslb-redis ~]# redis-cli -n 1 --pipe < 1.resp Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 100859 [root@mtl-gslb-redis ~]# redis-cli -n 2 --pipe < 2.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 1166 [root@mtl-gslb-redis ~]# redis-cli -n 3 --pipe < 3.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 8411 [root@mtl-gslb-redis ~]# redis-cli -n 4 --pipe < 4.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 456668 [root@mtl-gslb-redis ~]# redis-cli -n 5 --pipe < 5.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 1183 [root@mtl-gslb-redis ~]# redis-cli -n 6 --pipe < 6.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 6423 [root@mtl-gslb-redis ~]# redis-cli -n 8 --pipe < 8.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 2 [root@mtl-gslb-redis ~]# redis-cli -n 12 --pipe < 12.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 156

5. get redis info

Keyspace

db0:keys=6890,expires=0,avg_ttl=0 db1:keys=88555,expires=0,avg_ttl=0 The number of key is the same. db2:keys=1165,expires=0,avg_ttl=0 db3:keys=8410,expires=0,avg_ttl=0 db4:keys=456667,expires=0,avg_ttl=0 db5:keys=1182,expires=0,avg_ttl=0 db6:keys=6422,expires=0,avg_ttl=0 db8:keys=1,expires=0,avg_ttl=0 db12:keys=155,expires=0,avg_ttl=0

[linydquantil]

@yannh @mpartio
i use v0.5.0 to do some tests. this tool is good for me Next, I will do a more detailed test. thank you for your help

[linydquantil]

how to get redis-dump-go version? Maybe it needs to support functions like this: ./redis-dump-go --version.

[yannh]

Yes, that would be nice, thanks for the suggestion :) Might work on this at some point, would also accept a patch!

[linydquantil]

@yannh i found the same error when the redis version is 6.2.5

redis version: 6.2.5
redis dump go: v0.5.1

[cdb_dev_1109]# redis-cli -n 1 --pipe <  cdb_db_1.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
ERR Protocol error: invalid multibulk length

can you help me check it ?

[yannh]

Hi @linydquantil , I found this bug https://forums.servicestack.net/t/protocol-error-invalid-multibulk-length/4595 ? I am not sure this is related to redis-dump-go?