Don’t update the database version with the tools like ldb or sst_dump.

[ytrezq]

Expected behavior

The official Administration tools in this repository tries to keep database version.

Database administration shouldn’t means forced upgrade.

Actual behavior

even for just reading a single key without writing anything the footer version is updated which results in this error (because some software like OpenEthereum still use a RocksDb version which is several years behind and can’t be updated without a huge man power investment)

/usr/share/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/parity-rocksdb-sys-0.5.6/rocksdb/db/db_impl_compaction_flush.cc:1953] Compaction error: Corruption: Unknown Footer version. Maybe this file was created with newer version of RocksDB?

and prevents to use the old software.

[riversand963]

Let me guess the steps to reproduce this issue.

1. You have a database generated and accessed by an old RocksDB release.
2. You use ldb tool which is new to open the db.
3. Trying to access the db with old RocksDB failed.

Can you confirm whether this is the case?

[ytrezq]

@riversand963 more exactly, I synced a full archival node using OpenEthereum which is a process taking 1 or 2 year to complete for building the database. OpenEthereum is using an old api and thus an old release of Rocksdb, and this is likely to never get updated.

Then using ldb simply for reading updates the format and OpenEthereum can no longer use the database.

[riversand963]

If WAL is enabled, then it is possible that the command (such as 'query') you used to read from the db tries to open the db, recovers data from the WAL, inserts them to memtable, and flushes. This can generate a new SST file, whose format may be too new for your old RocksDB release. Ldb commands, get or scan seem fine because they open the db as read-only and does not do flush.

[ytrezq]

Even then I d like to do things like adding sst files which require newer version of ldb while still using the resulting database with OpenEthereum.

[ajkr]

Do you have a copy of the original DB to experiment with? If so, can you show the format_version from the OPTIONS file in the DB directory?

$ grep format_version OPTIONS-*
  format_version=4

Then using the newer ldb you can try passing --try_load_options, which should pick up the older format_version from the OPTIONS file, and then should only write files compatible with the older RocksDB version.

[ajkr]

Even then I d like to do things like adding sst files which require newer version of ldb while still using the resulting database with OpenEthereum.

FYI, I suspect we do not have good safeguards in place for calling new features like ldb ingest_extern_sst while expecting them to be readable by an old DB. I wouldn't be surprised if those write manifest entries that are unreadable by older versions...

[ytrezq]

@ajkr : yes, I do have a copy of the original database and yes, I tried using --try_load_options since an option file is present. The underlying Rocksdb version is 5.14.3. But after all, even the HISTORY file contains change which are said to not be optional and not forward compatible.