Closed posix4e closed 2 years ago
This error indicates that it's unable to parse the RDB file, so it can't skip this and keep reading the file.
Please describe how you created this RDB, which version of redis, and which module. and if possible upload an example RDB that reproduces this.
I believe the issue is our custom rdb. Once I delete the custom rdb keys it goes away. We used https://github.com/kristoff-it/redis-cuckoofilter+
@posix4e what do you mean by custom rdb? you mean a module data type (using redis module API)? these should be encoded in a way that they can be skipped. which version of redis do you use and which APIs?
bitnami/redis:5.0.5-ol-7-r149 with https://github.com/kristoff-it/redis-cuckoofilter
Once I removed the key I created with the cuckoofilter it was fixed. I'd assume that if I skipped the key with -k it would't crash but that didn't make a different.ce
Alex Newman 404.507.6749
On Sat, Nov 16, 2019 at 11:06 PM Oran Agra notifications@github.com wrote:
@posix4e https://github.com/posix4e what do you mean by custom rdb? you mean a module data type (using redis module API)? these should be encoded in a way that they can be skipped. which version of redis do you use and which APIs?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sripathikrishnan/redis-rdb-tools/issues/160?email_source=notifications&email_token=AAGE3WZE3L4ETBWRLXTUN3DQUDUOVA5CNFSM4JOC4TZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEIFQ3Y#issuecomment-554719343, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGE3W7GRTG2TUUCIDC53HDQUDUOVANCNFSM4JOC4TZA .
@kristoff-it any ideas why your crazy module causes that? want to help us get to the bottom of this?
IIRC i tested redis-rdb-tools with rejson keys and it worked. i assume that rdb isn't corrupted and can be read back by redis. can redis-check-rdb parse this rdb file? can you maybe post a very small rdb that reproduce it?
I just tested on my computer loading the latest released version of my module.
I created a couple of keys, added stuff to them, called SAVE
and then rebooted the server.
All worked fine, keys got loaded back in and the content seemed to be properly preserved.
Here is where I make use of the custom type API: https://github.com/kristoff-it/redis-cuckoofilter/blob/4c617941c529d8f9a59bf4562883c792607eaef8/src/t_cuckoofilter.zig#L119-L128
I don't really do anything non-standard AFAIK. I implement 3 different data types (one for each fingerprint size the module supports), and that's pretty much it.
I also tried to parse my rdb file with rdbtools (freshly installed from pip) and I got no errors.
If you create a few keys using the module..ie follow the example in the readme and you should be able to crash it. It's because it doesn't know how wide the types are I assume.
@posix4e this is what I did before replying yesterday. rdb-tools doesn't need to know about my module specifically to be able to parse the RDB file.
Can you please make sure you're using the latest release of redis-cuckoofilter (v1.1.0) and try to reproduce the problem in a RDB file to share with us, so that @oranagra can take a look?
I haven't been able to reproduce it on my own.
I'm sure I'm using the latest version
It seems as though no matter what keys I filter for, since I loaded a redis with a custom rdb there's no way to analyze this file with your parser. I'm curious if you could provide a mode where it's automatically skipped like rdb --command protocolskiperrors
Example of failed command rdb --command protocol -k 'anything' customrdb > /dev/null