search

speedb-io / speedb

A RocksDB compliant high performance scalable embedded key-value store
https://www.speedb.io/
Apache License 2.0
898 stars 67 forks source link

RocksDB Rebase: Rebase on RocksDB 8.6.7 #736

Closed udi-speedb closed 8 months ago

udi-speedb commented 9 months ago

@erez-speedb - Please run performance tests on branch 736-rocksdb-rebase-rebase-on-rocksdb-867 which is the candidate branch containing all of our commits in our main repo, rebased on RocksDB 8.6.7 The expectation is that we will not have performance degradation relative to our current main, that is rebased on RocksDB 8.1.1 Thanks

erez-speedb commented 9 months ago

There is an error when opening the DB 2023/11/26-16:28:48.010668 35869 [/version_set.cc:5976] Column family [default] (ID 0), log number is 2559 2023/11/26-16:28:48.010725 35869 [/db_impl/db_impl_open.cc:659] DB ID: 7912fb82-3916-4c16-914b-55e2c1878e33 2023/11/26-16:28:48.012105 35869 EVENT_LOG_v1 {"time_micros": 1701016128012098, "job": 1, "event": "recovery_started", "wal_files": [2559]} 2023/11/26-16:28:48.012126 35869 [/db_impl/db_impl_open.cc:1152] Recovering log #2559 mode 2 2023/11/26-16:28:48.292100 35869 [WARN] [/db_impl/db_impl_open.cc:1715] [default] [JOB 1] Level-0 flush during recover: Expected 424765 entries in memtable, but read 0 2023/11/26-16:28:48.292140 35869 EVENT_LOG_v1 {"time_micros": 1701016128292137, "job": 1, "event": "recovery_finished"} 2023/11/26-16:28:48.433198 35869 [WARN] [/db_impl/db_impl_open.cc:2240] DB::Open() failed: Corruption: Expected 424765 entries in memtable, but read 0 2023/11/26-16:28:48.433241 35869 [/db_impl/db_impl.cc:544] Shutdown: canceling all background work 2023/11/26-16:28:48.514578 35869 [/db_impl/db_impl.cc:742] Shutdown complete

udi-speedb commented 9 months ago

I think this is related to the following PR introduced in release 8.6.0. of RocksDB: https://github.com/facebook/rocksdb/pull/11611

I am easily able to reproduce using db_bench when the memtablerep is hash_spdb (the default). Can't reproduce when the memtablerep is skip_list. Reproduction:

./db_bench -benchmarks fillrandom -duration 5
./db_bench -benchmarks readrandom -duration 5 -use_existing_db

The DB fails to open. The code in DBImpl::WriteLevel0TableForRecovery() contains the new validation that fails. And the problem is reproduced on latest main when I cherry pick the commit abovementioned.

Yuval-Ariel commented 8 months ago

main at 5e356db3a709003ffd2492cdffd766ffbc5fbf0d is the result of this rebase