search

sripathikrishnan / redis-rdb-tools

Parse Redis dump.rdb files, Analyze Memory, and Export Data to JSON
https://rdbtools.com
MIT License
5.07k stars 739 forks source link

Unknown opcode in custom rdbs #160

Closed posix4e closed 2 years ago

posix4e commented 4 years ago

It seems as though no matter what keys I filter for, since I loaded a redis with a custom rdb there's no way to analyze this file with your parser. I'm curious if you could provide a mode where it's automatically skipped like rdb --command protocolskiperrors

Example of failed command rdb --command protocol -k 'anything' customrdb > /dev/null 

Traceback (most recent call last):
  File "/home/posix4e/.local/bin/rdb", line 11, in <module>
    sys.exit(main())
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/cli/rdb.py", line 101, in main
    parser.parse(dump_file)
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 394, in parse
    self.parse_fd(open(filename, "rb"))
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 463, in parse_fd
    self.skip_object(f, data_type)
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 656, in skip_object
    self.skip_module(f)
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 834, in skip_module
    raise Exception("Unknown module opcode %s" % opcode)
Exception: Unknown module opcode 8198801549535663106
oranagra commented 4 years ago

This error indicates that it's unable to parse the RDB file, so it can't skip this and keep reading the file.

Please describe how you created this RDB, which version of redis, and which module. and if possible upload an example RDB that reproduces this.

posix4e commented 4 years ago

I believe the issue is our custom rdb. Once I delete the custom rdb keys it goes away. We used https://github.com/kristoff-it/redis-cuckoofilter+

oranagra commented 4 years ago

@posix4e what do you mean by custom rdb? you mean a module data type (using redis module API)? these should be encoded in a way that they can be skipped. which version of redis do you use and which APIs?

posix4e commented 4 years ago

bitnami/redis:5.0.5-ol-7-r149 with https://github.com/kristoff-it/redis-cuckoofilter

Once I removed the key I created with the cuckoofilter it was fixed. I'd assume that if I skipped the key with -k it would't crash but that didn't make a different.ce

Alex Newman 404.507.6749

On Sat, Nov 16, 2019 at 11:06 PM Oran Agra notifications@github.com wrote:

@posix4e https://github.com/posix4e what do you mean by custom rdb? you mean a module data type (using redis module API)? these should be encoded in a way that they can be skipped. which version of redis do you use and which APIs?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sripathikrishnan/redis-rdb-tools/issues/160?email_source=notifications&email_token=AAGE3WZE3L4ETBWRLXTUN3DQUDUOVA5CNFSM4JOC4TZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEIFQ3Y#issuecomment-554719343, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGE3W7GRTG2TUUCIDC53HDQUDUOVANCNFSM4JOC4TZA .

itamarhaber commented 4 years ago

@kristoff-it any ideas why your crazy module causes that? want to help us get to the bottom of this?

oranagra commented 4 years ago

IIRC i tested redis-rdb-tools with rejson keys and it worked. i assume that rdb isn't corrupted and can be read back by redis. can redis-check-rdb parse this rdb file? can you maybe post a very small rdb that reproduce it?

kristoff-it commented 4 years ago

I just tested on my computer loading the latest released version of my module. I created a couple of keys, added stuff to them, called SAVE and then rebooted the server. All worked fine, keys got loaded back in and the content seemed to be properly preserved.

Here is where I make use of the custom type API: https://github.com/kristoff-it/redis-cuckoofilter/blob/4c617941c529d8f9a59bf4562883c792607eaef8/src/t_cuckoofilter.zig#L119-L128

I don't really do anything non-standard AFAIK. I implement 3 different data types (one for each fingerprint size the module supports), and that's pretty much it.

I also tried to parse my rdb file with rdbtools (freshly installed from pip) and I got no errors.

posix4e commented 4 years ago

If you create a few keys using the module..ie follow the example in the readme and you should be able to crash it. It's because it doesn't know how wide the types are I assume.

kristoff-it commented 4 years ago

@posix4e this is what I did before replying yesterday. rdb-tools doesn't need to know about my module specifically to be able to parse the RDB file.

Can you please make sure you're using the latest release of redis-cuckoofilter (v1.1.0) and try to reproduce the problem in a RDB file to share with us, so that @oranagra can take a look?

I haven't been able to reproduce it on my own.

posix4e commented 4 years ago

I'm sure I'm using the latest version