search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
27.83k stars 6.2k forks source link

Forward Compatibility Errors #12819

Open rk210 opened 1 week ago

rk210 commented 1 week ago

Forward compatibility Errors

Our services use rocksDB v5.7.2, and we are looking to upgrade to rocksDB v8.3.2. However, our service is stateful, and receives continuous writes. We would like to accommodate for a rollback, but rollback fails with forward compatibility errors such as:

  1. Corruption: VersionEdit: unknown tag
  2. Invalid argument: Can't parse BlockBasedTableOptions:: checksum
  3. Corruption: unknown checksum type 4

We were looking into making some code changes to allow for forward compatibility. This includes changing the format_version to 2, and also changing the v8 checksum to be kCRC32c by default. However, we would still face issues with reverting the patch to allow for forward compatibility. Is there a way we can guarantee forward compatibility? How do we upgrade rocksDB while allowing for rollbacks?

ajkr commented 5 days ago

We don't have proper data format versioning and therefore lack clear downgrade rules. It is one thing I hope to change soon. In the meantime the best chance for success is to not downgrade more than a few minor versions within a major version, or one minor version across major versions. 5.7 <-> 8.3 includes tens of minor versions and three major version boundaries so isn't likely to work.