Test downgrade - Githubissues

basho / bitcask

because you need another a key/value storage engine

1.29k stars 173 forks source link

Test downgrade #178

Closed engelsanchez closed 10 years ago

engelsanchez commented 10 years ago

The 1.7.0 version now has different file formats:

Hint files now use a bit in the size field to signal a tombstone. To old code, it would look like a massive file offset and therefore a corrupted hint file. Users would likely have to delete hint files in case of a downgrade.
Data files now contain 2 new tombstone formats. To Riak, they would look like invalid objects, which would hopefully always end up as a not found.

We need to verify that there is a path to downgrade, however annoying.

/cc @jonmeredith

jonmeredith commented 10 years ago

Assigned to 2.0 final - thanks Engel.

engelsanchez commented 10 years ago

Well well. The result of my testing was unexpected: Downgrade is not possible, but it has nothing to do with the tombstone changes mentioned here. The work done to support a more efficient form of key encoding makes the new keys unreadable by the older code. I believe related to this commit: https://github.com/basho/riak_kv/commit/23403462f856fc5a7916923f7aa6a3cd3466f753#diff-709ab916aa28dc2a7f28acd8874529dd

I'm unable to read values whether they were deleted in 2.0 or not. Also, when listing keys the workers crash trying to b2t the keys:

2014-08-27 18:27:01.830 [error] <0.608.0> CRASH REPORT Process <0.608.0> with 0 neighbours
 exited with reason: bad argument in call to erlang:binary_to_term(<<2,0,1,98,49,56>>) in 
riak_kv_bitcask_backend:'-fold_keys_fun/2-fun-1-'/4 line 461 in gen_server:terminate/6 line 747

So, as we stand now, there is no easy downgrade path without some kind of tool to reformat all the data :(

engelsanchez commented 10 years ago

Although I laughed at this option once, maybe we should do a release in the 1.4 series that can read this new format. I have now seen that approach taken with other NoSQL DBs, although it's less than optimal.

evanmcc commented 10 years ago

Let me know if there's anything that I can do to do help.

It's also possible to just not use that code, I think just with a config change, or an easy code change.

On Wed, Aug 27, 2014 at 3:44 PM, Engel A. Sanchez notifications@github.com wrote:

Although I laughed at this option once, maybe we should do a release in the 1.4 series that can read this new format. I have now seen that approach taken with other NoSQL DBs, although it's less than optimal.

— Reply to this email directly or view it on GitHub https://github.com/basho/bitcask/issues/178#issuecomment-53651423.

engelsanchez commented 10 years ago

2.0.0 released with this downgrade problem. A script is available for people who need to downgrade back to an old version after moving to 2.0.0.