sripathikrishnan / redis-rdb-tools

Parse Redis dump.rdb files, Analyze Memory, and Export Data to JSON
https://rdbtools.com
MIT License
5.07k stars 739 forks source link

Number of keys mismatch #162

Open siva-7112 opened 4 years ago

siva-7112 commented 4 years ago

Tried generating memory report but large number of keys are missing in memory.csv

oranagra commented 4 years ago

@siva-7112 can you share more details? which version of rdbtools you're using and which version of redis? what are the types of keys missing (maybe they're all of the same type)?

maybe these are all volatile keys? redis < 5 doesn't write expired keys into the rdb, so they could still show up in DBSIZE and INFO KEYSPACE, but will be missing from the rdb. can you try loading this rdb back into redis and see if the same number of keys are gone...

siva-7112 commented 4 years ago

@oranagra rdbtools - 0.1.4 redis version : 5.0.3

Total Number of keys in redis :
db0:keys=1643582,expires=0,avg_ttl=0 db1:keys=753355,expires=359421,avg_ttl=797119291 db2:keys=73837,expires=5355,avg_ttl=28691183 db15:keys=551540,expires=0,avg_ttl=0

After parsing using rdbtools for memory report total number of keys : 425639 After loading same dump in different redis the number of keys :

db0:keys=1643582,expires=0,avg_ttl=0 db1:keys=753155,expires=359221,avg_ttl=774841181 db2:keys=73837,expires=5355,avg_ttl=32687121 db15:keys=551540,expires=0,avg_ttl=0

As you can see the number of keys missing in memory report However, Total redis size : 2.1 GB sum of 'size_in_bytes' in memory report is - 1.89 GB

it seems the total memory is closely matching but the number of key counts have more difference.

oranagra commented 4 years ago

@siva-7112 i guess we must be able to reproduce in order to investigate. it would be best if you can narrow down the scenario with some trial and error and find what types of keys are missing, so that you can upload an example rdb with just a few small keys that reproduce the problem. while you're at it, feel free to fix the bug and issue a PR too 8-)