search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
28.45k stars 6.29k forks source link

LoadLatestOptions fails with "Missing configurable object: memtable_factory" on RDB created with <=6.24 after RDB ver 6.25 with g++ 4.8.5 #9225

Closed thatsafunnyname closed 2 years ago

thatsafunnyname commented 2 years ago

Hello and thanks for RocksDB,

LoadLatestOptions (and CheckOptionsCompatibility) fails with "Missing configurable object: memtable_factory" on RDB created with ver <=6.24 after RDB ver 6.25

In particular the behaviour changes with commit https://github.com/facebook/rocksdb/commit/beed86473a078f58e6a8f214068a142bfc53d6aa for https://github.com/facebook/rocksdb/pull/8419

Command line reproducer:

Create a RDB with version 6.24.2 code:

> ~/rocksdb_builds/rocksdb-6.24.2/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_24_2
OK

Check the version in the options file:

> grep rocksdb_version /tmp/test_rocksdb_6_24_2/OPTIONS-000007
  rocksdb_version=6.24.2

Try and read the RDB using ldb from version 6.26.0 and command line option --try_load_options it will fail.

> ~/rocksdb_builds/rocksdb-6.26.0/ldb scan --db=/tmp/test_rocksdb_6_24_2/ --try_load_options
Failed: Invalid argument: Missing configurable object: memtable_factory

Main prog reproducer

Based on https://github.com/facebook/rocksdb/wiki/RocksDB-Options-File

#include <iostream>
#include <fstream>
#include <rocksdb/utilities/options_util.h>

int main(int argc, char **argv){
  // created using v6.24.2 using: ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_24_2
  std::string db_name("/tmp/test_rocksdb_6_24_2"); 

  rocksdb::DBOptions loaded_db_opt;
  std::vector<rocksdb::ColumnFamilyDescriptor> loaded_cf_descs;
  rocksdb::Status status = LoadLatestOptions(db_name,
                                             rocksdb::Env::Default(), &loaded_db_opt,
                                             &loaded_cf_descs);
  if (status.IsNotFound()) {
    std::cout << "Not found OK " << db_name << std::endl;
    return(0);
  }

  if (!status.ok()) {
    std::string msg("Failed to LoadLatestOptions from: " + db_name + ": " + status.ToString());
    throw std::runtime_error(msg);
  } else {
    std::cout << "OK LoadLatestOptions " << db_name << std::endl;
    return(0);
  }
}

When built and run against e40b04e (and before) it returns:

  OK LoadLatestOptions /tmp/test_rocksdb_6_24_2

when built and run against beed864 (and after) it returns:

  terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to LoadLatestOptions from: /tmp/test_rocksdb_6_24_2: Invalid argument: Missing configurable object: memtable_factory

This was noticed during a RocksDB upgrade when an integration test that uses a RDB from a previous old RDB version started to fail when running CheckOptionsCompatibility.

Thanks

In addition, after running the below:

> ~/rocksdb_builds/rocksdb-6.17.3/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_17_3
OK
> ~/rocksdb_builds/rocksdb-6.20.3/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_20_3
OK
> ~/rocksdb_builds/rocksdb-6.24.2/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_24_2
OK
> ~/rocksdb_builds/rocksdb-6.25.1/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_25_1
OK
> ~/rocksdb_builds/rocksdb-6.25.3/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_25_3
OK
> ~/rocksdb_builds/rocksdb-6.26.0/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_0
OK
> ~/rocksdb_builds/rocksdb-6.26.1/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_1
OK
> ~/rocksdb_builds/rocksdb-main/ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_27_0
OK

I noticed:

> ls -1 /tmp/test_rocksdb_6*/OP*
/tmp/test_rocksdb_6_17_3/OPTIONS-000006
/tmp/test_rocksdb_6_20_3/OPTIONS-000007
/tmp/test_rocksdb_6_24_2/OPTIONS-000007
/tmp/test_rocksdb_6_25_1/OPTIONS-000006.dbtmp
/tmp/test_rocksdb_6_25_3/OPTIONS-000006.dbtmp
/tmp/test_rocksdb_6_26_0/OPTIONS-000006.dbtmp
/tmp/test_rocksdb_6_26_1/OPTIONS-000006.dbtmp
/tmp/test_rocksdb_6_27_0/OPTIONS-000006.dbtmp
> grep 'Unnable to persist options' /tmp/test_rocksdb_6*/LOG
/tmp/test_rocksdb_6_25_1/LOG:2021/11/29-09:56:09.019822 7f51e44a8b00 [WARN] [/db_impl/db_impl.cc:4214] Unnable to persist options -- Invalid argument: Missing configurable object: memtable_factory
/tmp/test_rocksdb_6_25_3/LOG:2021/11/29-09:56:16.697449 7f714ff8eb00 [WARN] [/db_impl/db_impl.cc:4214] Unnable to persist options -- Invalid argument: Missing configurable object: memtable_factory
/tmp/test_rocksdb_6_26_0/LOG:2021/11/29-09:56:27.635650 7fc10ca1b340 [WARN] [/db_impl/db_impl.cc:4215] Unnable to persist options -- Invalid argument: Missing configurable object: memtable_factory
/tmp/test_rocksdb_6_26_1/LOG:2021/11/29-09:56:35.806378 7f7596a97b40 [WARN] [/db_impl/db_impl.cc:4215] Unnable to persist options -- Invalid argument: Missing configurable object: memtable_factory
/tmp/test_rocksdb_6_27_0/LOG:2021/11/29-09:57:04.379791 140249826884416 [WARN] [/db_impl/db_impl.cc:4216] Unnable to persist options -- Invalid argument: Missing configurable object: memtable_factory
akankshamahajan15 commented 2 years ago

@mrambacher Hi Mark, Do you more context about this.

ltamasi commented 2 years ago

@mrambacher Could you take a look at this?

ltamasi commented 2 years ago

@thatsafunnyname For the record, I could not reproduce this using the standard skiplist memtable factory.

Do you use a custom memtable factory by any chance? If so, you might want to update your implementation to fit the Customizable framework and register it with the ObjectRegistry (see how it's done for the built-in factories in #8419).

mrambacher commented 2 years ago

@thatsafunnyname I cannot reproduce this problem with an options file generated from 6.24.2. I have tried to reproduce this using 6.26, 6.27, and main without luck. Can you send me the OPTIONS file that fails? What is different than one that works? What is different between the failed (dbtmp) Options files and the ones that succeeded?

How are you building RocksDB? Are you using LITE mode or any other switches?

thatsafunnyname commented 2 years ago

Steps to reproduce, this is on a "Red Hat Enterprise Linux Server release 7.9 (Maipo)" host, 3.10.0-1160.31.1.el7.x86_64, sorry I did not test on another OS/distro or specify the OS/distro previously.

> wget 'https://github.com/facebook/rocksdb/archive/refs/tags/v6.26.1.tar.gz'
> gzip -d v6.26.1.tar.gz
> tar -xvf v6.26.1.tar
> cd rocksdb-6.26.1
> make -j12 USE_RTTI=1 LIB_MODE=shared DEBUG_LEVEL=0 ldb
> ./ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_1
> ls -1 /tmp/test_rocksdb_6_26_1/
000005.log
CURRENT
IDENTITY
LOCK
LOG
MANIFEST-000004
OPTIONS-000006.dbtmp
> grep 'Unnable to persist options' /tmp/test_rocksdb_6_26_1/LOG
2021/12/01-16:50:21.747051 7fb48ee08b40 [WARN] [/db_impl/db_impl.cc:4215] Unnable to persist options -- Invalid argument: Missing configurable object: memtable_factory

I tested without USE_RTTI=1 and it behaves the same way.

> cat make_config.mk
CC=cc
CXX=g++
AR=gcc-ar
PLATFORM=OS_LINUX
PLATFORM_LDFLAGS= -lpthread -lrt -ldl -lsnappy -lgflags -lz -llz4 -lzstd
PLATFORM_CMAKE_FLAGS=
JAVA_LDFLAGS= -lpthread -lrt -ldl -lsnappy -lz -llz4 -lzstd
JAVA_STATIC_LDFLAGS= -lpthread -lrt -ldl
JAVA_STATIC_DEPS_CCFLAGS=
JAVA_STATIC_DEPS_CXXFLAGS=
JAVA_STATIC_DEPS_LDFLAGS=
JAVAC_ARGS=-source 7
VALGRIND_VER=
PLATFORM_CCFLAGS= -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=1 -DGFLAGS_NAMESPACE=gflags -DZLIB -DLZ4 -DZSTD -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_BACKTRACE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_AUXV_GETAUXVAL_PRESENT -march=native   -DHAVE_SSE42  -DHAVE_PCLMUL  -DHAVE_AVX2  -DHAVE_BMI  -DHAVE_LZCNT -DHAVE_UINT128_EXTENSION -DROCKSDB_SUPPORT_THREAD_LOCAL
PLATFORM_CXXFLAGS=-std=c++11  -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=1 -DGFLAGS_NAMESPACE=gflags -DZLIB -DLZ4 -DZSTD -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_BACKTRACE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_AUXV_GETAUXVAL_PRESENT -march=native   -DHAVE_SSE42  -DHAVE_PCLMUL  -DHAVE_AVX2  -DHAVE_BMI  -DHAVE_LZCNT -DHAVE_UINT128_EXTENSION -DROCKSDB_SUPPORT_THREAD_LOCAL
PLATFORM_SHARED_CFLAGS=-fPIC
PLATFORM_SHARED_EXT=so
PLATFORM_SHARED_LDFLAGS=-Wl,--no-as-needed -shared -Wl,-soname -Wl,
PLATFORM_SHARED_VERSIONED=true
EXEC_LDFLAGS=-ldl
JEMALLOC_INCLUDE=
JEMALLOC_LIB=
ROCKSDB_MAJOR=6
ROCKSDB_MINOR=26
ROCKSDB_PATCH=1
CLANG_SCAN_BUILD=scan-build
CLANG_ANALYZER=
PROFILING_FLAGS=-pg
FIND=find
WATCH=watch
JEMALLOC=1
WITH_JEMALLOC_FLAG=1
LUA_PATH=
> g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)

Below is on a CentOS Linux release 8.4.2105 host, 4.18.0-305.19.1.el8_4.x86_64, it does not reproduce:

> wget 'https://github.com/facebook/rocksdb/archive/refs/tags/v6.26.1.tar.gz'
> gzip -d v6.26.1.tar.gz
> tar -xvf v6.26.1.tar
> cd rocksdb-6.26.1
> make -j12 USE_RTTI=1 LIB_MODE=shared DEBUG_LEVEL=0 ldb
> ./ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_1
OK
> ls -1 /tmp/test_rocksdb_6_26_1/
000005.log
CURRENT
IDENTITY
LOCK
LOG
MANIFEST-000004
OPTIONS-000007
> cat make_config.mk
CC=cc
CXX=g++
AR=gcc-ar
PLATFORM=OS_LINUX
PLATFORM_LDFLAGS= -lpthread -lrt -ldl -lsnappy -lgflags -lz -llz4 -lzstd
PLATFORM_CMAKE_FLAGS=
JAVA_LDFLAGS= -lpthread -lrt -ldl -lsnappy -lz -llz4 -lzstd
JAVA_STATIC_LDFLAGS= -lpthread -lrt -ldl
JAVA_STATIC_DEPS_CCFLAGS=
JAVA_STATIC_DEPS_CXXFLAGS=
JAVA_STATIC_DEPS_LDFLAGS=
JAVAC_ARGS=-source 7
VALGRIND_VER=
PLATFORM_CCFLAGS= -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=1 -DZLIB -DLZ4 -DZSTD -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_BACKTRACE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_AUXV_GETAUXVAL_PRESENT -march=native   -DHAVE_SSE42  -DHAVE_PCLMUL  -DHAVE_AVX2  -DHAVE_BMI  -DHAVE_LZCNT -DHAVE_UINT128_EXTENSION -DROCKSDB_SUPPORT_THREAD_LOCAL
PLATFORM_CXXFLAGS=-std=c++11  -faligned-new -DHAVE_ALIGNED_NEW -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=1 -DZLIB -DLZ4 -DZSTD -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_BACKTRACE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_AUXV_GETAUXVAL_PRESENT -march=native   -DHAVE_SSE42  -DHAVE_PCLMUL  -DHAVE_AVX2  -DHAVE_BMI  -DHAVE_LZCNT -DHAVE_UINT128_EXTENSION -DROCKSDB_SUPPORT_THREAD_LOCAL
PLATFORM_SHARED_CFLAGS=-fPIC
PLATFORM_SHARED_EXT=so
PLATFORM_SHARED_LDFLAGS=-Wl,--no-as-needed -shared -Wl,-soname -Wl,
PLATFORM_SHARED_VERSIONED=true
EXEC_LDFLAGS=-ldl
JEMALLOC_INCLUDE=
JEMALLOC_LIB=
ROCKSDB_MAJOR=6
ROCKSDB_MINOR=26
ROCKSDB_PATCH=1
CLANG_SCAN_BUILD=scan-build
CLANG_ANALYZER=
PROFILING_FLAGS=-pg
FIND=find
WATCH=watch
JEMALLOC=1
WITH_JEMALLOC_FLAG=1
LUA_PATH=
> g++ --version
g++ (GCC) 8.4.1 20200928 (Red Hat 8.4.1-1)
thatsafunnyname commented 2 years ago

When I use DEBUG_LEVEL=2 on the RHEL7 host an assert fails (it does not fail on the CentOS 8 host):

> make -j12 LIB_MODE=shared DEBUG_LEVEL=2 ldb
> ./ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_1
ldb: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
Aborted
(gdb) bt
#0  0x00007ffff41ea387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff41eba78 in __GI_abort () at abort.c:90
#2  0x00007ffff41e31a6 in __assert_fail_base (fmt=0x7ffff433ee00 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x7ffff74fe2d2 "false", file=file@entry=0x7ffff74fe2c4 "util/regex.cc", line=line@entry=29,
    function=function@entry=0x7ffff74fe420 <rocksdb::Regex::Matches(std::string const&) const::__PRETTY_FUNCTION__> "bool rocksdb::Regex::Matches(const string&) const") at assert.c:92
#3  0x00007ffff41e3252 in __GI___assert_fail (assertion=0x7ffff74fe2d2 "false", file=0x7ffff74fe2c4 "util/regex.cc", line=29,
    function=0x7ffff74fe420 <rocksdb::Regex::Matches(std::string const&) const::__PRETTY_FUNCTION__> "bool rocksdb::Regex::Matches(const string&) const")
    at assert.c:101
#4  0x00007ffff7296d1c in rocksdb::Regex::Matches (this=0x7ffff3938510, str="SkipListFactory") at util/regex.cc:29
#5  0x00007ffff7244ab3 in rocksdb::ObjectLibrary::FactoryEntry<rocksdb::MemTableRepFactory>::matches (this=0x7ffff3938500, target="SkipListFactory")
    at ./include/rocksdb/utilities/object_registry.h:79
#6  0x00007ffff735a8ab in rocksdb::ObjectLibrary::FindEntry (this=0x7ffff382f398, type="MemTableRepFactory", name="SkipListFactory")
    at utilities/object_registry.cc:24
#7  0x00007ffff735aefb in rocksdb::ObjectRegistry::FindEntry (this=0x7ffff3848300, type="MemTableRepFactory", name="SkipListFactory")
    at utilities/object_registry.cc:95
#8  0x00007ffff735af9d in rocksdb::ObjectRegistry::FindEntry (this=0x7ffff3848618, type="MemTableRepFactory", name="SkipListFactory")
    at utilities/object_registry.cc:102
#9  0x00007ffff724473a in rocksdb::ObjectRegistry::NewObject<rocksdb::MemTableRepFactory> (this=0x7ffff3848618, target="SkipListFactory",
    guard=0x7fffffff7750, errmsg=0x7fffffff7500) at ./include/rocksdb/utilities/object_registry.h:181
#10 0x00007ffff724447f in rocksdb::ObjectRegistry::NewUniqueObject<rocksdb::MemTableRepFactory> (this=0x7ffff3848618, target="SkipListFactory",
    result=0x7fffffff7750) at ./include/rocksdb/utilities/object_registry.h:201
#11 0x00007ffff7242a6e in rocksdb::NewUniqueObject<rocksdb::MemTableRepFactory> (config_options=..., id="SkipListFactory", opt_map=
    std::unordered_map with 0 elements, result=0x7fffffff7750) at ./include/rocksdb/utilities/customizable_util.h:249
#12 0x00007ffff7242324 in rocksdb::MemTableRepFactory::CreateFromString (config_options=..., value="SkipListFactory", result=0x7fffffff7750)
    at table/plain/plain_table_factory.cc:267
#13 0x00007ffff714c533 in rocksdb::__lambda20::operator() (__closure=0x7ffff384b1f8, opts=..., value="SkipListFactory", addr=0x7ffff38d3688)
    at options/cf_options.cc:595
#14 0x00007ffff714f15e in std::_Function_handler<rocksdb::Status(const rocksdb::ConfigOptions&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, void*), rocksdb::__lambda20>::_M_invoke(const std::_Any_data &, const rocksdb::ConfigOptions &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, void *) (__functor=..., __args#0=..., __args#1="memtable_factory", __args#2="SkipListFactory", __args#3=0x7ffff38d3688)
    at /usr/include/c++/4.8.2/functional:2057
#15 0x00007ffff7189f02 in std::function<rocksdb::Status (rocksdb::ConfigOptions const&, std::string const&, std::string const&, void*)>::operator()(rocksdb::ConfigOptions const&, std::string const&, std::string const&, void*) const (this=0x7ffff38a0f38, __args#0=..., __args#1="memtable_factory",
    __args#2="SkipListFactory", __args#3=0x7ffff38d3688) at /usr/include/c++/4.8.2/functional:2471
#16 0x00007ffff7182e70 in rocksdb::OptionTypeInfo::Parse (this=0x7ffff38a0f30, config_options=..., opt_name="memtable_factory", value="SkipListFactory",
    opt_ptr=0x7ffff38d35d0) at options/options_helper.cc:886
#17 0x00007ffff715f270 in rocksdb::Configurable::ParseOption (this=0x7ffff38d3400, config_options=..., opt_info=..., opt_name="memtable_factory",
    opt_value="SkipListFactory", opt_ptr=0x7ffff38d35d0) at options/configurable.cc:268
#18 0x00007ffff715ff55 in rocksdb::ConfigurableHelper::ConfigureCustomizableOption (config_options=..., configurable=..., opt_info=...,
    opt_name="memtable_factory", name="memtable_factory", value="SkipListFactory", opt_ptr=0x7ffff38d35d0) at options/configurable.cc:415
#19 0x00007ffff716075f in rocksdb::ConfigurableHelper::ConfigureOption (config_options=..., configurable=..., opt_info=..., opt_name="memtable_factory",
    name="memtable_factory", value="SkipListFactory", opt_ptr=0x7ffff38d35d0) at options/configurable.cc:482
#20 0x00007ffff715f7b1 in rocksdb::ConfigurableHelper::ConfigureSomeOptions (config_options=..., configurable=...,
    type_map=std::unordered_map with 28 elements = {...}, options=0x7fffffff7ef0, opt_ptr=0x7ffff38d35d0) at options/configurable.cc:345
#21 0x00007ffff715f3ad in rocksdb::ConfigurableHelper::ConfigureOptions (config_options=..., configurable=...,
    opts_map=std::unordered_map with 60 elements = {...}, unused=0x0) at options/configurable.cc:283
#22 0x00007ffff715eb4c in rocksdb::Configurable::ConfigureOptions (this=0x7ffff38d3400, config_options=...,
    opts_map=std::unordered_map with 60 elements = {...}, unused=0x0) at options/configurable.cc:181
#23 0x00007ffff7154de4 in rocksdb::ConfigurableCFOptions::ConfigureOptions (this=0x7ffff38d3400, config_options=...,
    opts_map=std::unordered_map with 60 elements = {...}, unused=0x0) at options/cf_options.cc:750
#24 0x00007ffff715ea2c in rocksdb::Configurable::ConfigureFromMap (this=0x7ffff38d3400, config_options=...,
    opts_map=std::unordered_map with 60 elements = {...}, unused=0x0) at options/configurable.cc:155
#25 0x00007ffff715e9e4 in rocksdb::Configurable::ConfigureFromMap (this=0x7ffff38d3400, config_options=...,
    opts_map=std::unordered_map with 60 elements = {...}) at options/configurable.cc:147
#26 0x00007ffff7189954 in rocksdb::ConfigureFromMap<rocksdb::ColumnFamilyOptions> (config_options=..., opt_map=std::unordered_map with 60 elements = {...},
    option_name="ColumnFamilyOptions", config=0x7ffff38d3400, new_opts=0x7ffff38caf00) at options/options_helper.cc:536
#27 0x00007ffff7181bb1 in rocksdb::GetColumnFamilyOptionsFromMap (config_options=..., base_options=..., opts_map=std::unordered_map with 60 elements = {...},
    new_options=0x7ffff38caf00) at options/options_helper.cc:657
#28 0x00007ffff71993ea in rocksdb::RocksDBOptionsParser::EndSection (this=0x7fffffffac60, config_options=..., section=rocksdb::kOptionSectionCFOptions,
    section_title="CFOptions", section_arg="default", opt_map=std::unordered_map with 60 elements = {...}) at options/options_parser.cc:439
#29 0x00007ffff71983dd in rocksdb::RocksDBOptionsParser::Parse (this=0x7fffffffac60, config_options_in=...,
    file_name="/tmp/test_rocksdb_6_26_1/OPTIONS-000006.dbtmp", fs=0x7ffff7aa3a00 <rocksdb::FileSystem::Default()::default_fs>) at options/options_parser.cc:281
#30 0x00007ffff7199ceb in rocksdb::RocksDBOptionsParser::VerifyRocksDBOptionsFromFile (config_options_in=..., db_opt=...,
    cf_names=std::vector of length 1, capacity 1 = {...}, cf_opts=std::vector of length 1, capacity 1 = {...},
    file_name="/tmp/test_rocksdb_6_26_1/OPTIONS-000006.dbtmp", fs=0x7ffff7aa3a00 <rocksdb::FileSystem::Default()::default_fs>) at options/options_parser.cc:562
#31 0x00007ffff7197084 in rocksdb::PersistRocksDBOptions (config_options_in=..., db_opt=..., cf_names=std::vector of length 1, capacity 1 = {...},
    cf_opts=std::vector of length 1, capacity 1 = {...}, file_name="/tmp/test_rocksdb_6_26_1/OPTIONS-000006.dbtmp",
    fs=0x7ffff7aa3a00 <rocksdb::FileSystem::Default()::default_fs>) at options/options_parser.cc:141
#32 0x00007ffff719609b in rocksdb::PersistRocksDBOptions (db_opt=..., cf_names=std::vector of length 1, capacity 1 = {...},
    cf_opts=std::vector of length 1, capacity 1 = {...}, file_name="/tmp/test_rocksdb_6_26_1/OPTIONS-000006.dbtmp",
    fs=0x7ffff7aa3a00 <rocksdb::FileSystem::Default()::default_fs>) at options/options_parser.cc:53
#33 0x00007ffff6e9ec6a in rocksdb::DBImpl::WriteOptionsFile (this=0x7ffff38e3000, need_mutex_lock=false, need_enter_write_thread=false)
    at db/db_impl/db_impl.cc:4201
#34 0x00007ffff6f288bf in rocksdb::DBImpl::Open (db_options=..., dbname="/tmp/test_rocksdb_6_26_1",
    column_families=std::vector of length 1, capacity 1 = {...}, handles=0x7fffffffce60, dbptr=0x7ffff38b5640, seq_per_batch=false, batch_per_txn=true)
    at db/db_impl/db_impl_open.cc:1768
#35 0x00007ffff6f27043 in rocksdb::DB::Open (db_options=..., dbname="/tmp/test_rocksdb_6_26_1", column_families=std::vector of length 1, capacity 1 = {...},
    handles=0x7fffffffce60, dbptr=0x7ffff38b5640) at db/db_impl/db_impl_open.cc:1530
#36 0x00007ffff6f26de7 in rocksdb::DB::Open (options=..., dbname="/tmp/test_rocksdb_6_26_1", dbptr=0x7ffff38b5640) at db/db_impl/db_impl_open.cc:1507
#37 0x00007ffff7b52427 in rocksdb::LDBCommand::OpenDB (this=0x7ffff38b5600) at tools/ldb_cmd.cc:429
#38 0x00007ffff7b51704 in rocksdb::LDBCommand::Run (this=0x7ffff38b5600) at tools/ldb_cmd.cc:314
#39 0x00007ffff7b9cbe1 in rocksdb::LDBCommandRunner::RunCommand (argc=6, argv=0x7fffffffe318, options=..., ldb_options=..., column_families=0x0)
    at tools/ldb_tool.cc:142
#40 0x00007ffff7b9cdc8 in rocksdb::LDBTool::Run (this=0x7fffffffdcdf, argc=6, argv=0x7fffffffe318, options=..., ldb_options=..., column_families=0x0)
    at tools/ldb_tool.cc:156
#41 0x0000000000404da4 in main (argc=6, argv=0x7fffffffe318) at tools/ldb.cc:12
pdillinger commented 2 years ago

I'm guessing that you would see numerous failures if running RocksDB unit tests (make -j12 check).

This API unfortunately depends on std::regex which we know is a bad state to be in https://github.com/facebook/rocksdb/blob/main/include/rocksdb/utilities/regex.h#L23

And it looks like old GCC have incomplete / buggy std::regex. https://stackoverflow.com/a/12665408/454544

mrambacher commented 2 years ago

My guess is that the Regex::Parse in include/rocksb/utilities/object_registry.h is failing. Can you print out the status if it (is not OK) for an LDB run and send me the ones that fail? It might be possible to restructure some of the code to avoid this issue if I knew which regex were causing the problem...

pdillinger commented 2 years ago

8419 introduced the (:[0-9]*)? part of regex matching

thatsafunnyname commented 2 years ago

RocksDB unit tests do show numerous failures when on the RHEL7 host with g++ 4.8.5:

> make -j12 check
> grep -i 'Assertion.*failed' t/log* | head
t/log-deletefile_test:deletefile_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-auto_roll_logger_test-AutoRollLoggerTest.LogFileExistence:auto_roll_logger_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.BackgroundThreadCpuPriority:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.BackupOptions:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.BackupWithMetadata:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.BinaryMetadata:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.BlobFileCorruptedBeforeBackup:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.ChangeManifestDuringBackupCreation:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.Concurrency:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
t/log-run-backupable_db_test-BackupEngineTest.CorruptFileMaintainSize:backupable_db_test: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.

It looks like the build-linux-gcc-4_8-no_test_run CI test job in .circleci/config.yml is not running tests, and the make-gcc4.8 in .travis.yml is also not.

With the following change in rocksdb-6.26.1/include/rocksdb/utilities/object_registry.h:

  class FactoryEntry : public Entry {
   public:
    FactoryEntry(const std::string& name, FactoryFunc<T> f)
        : Entry(name), factory_(std::move(f)) {
      // FIXME: the API needs to expose this failure mode. For now, bad regexes
      // will match nothing.
      std::cout << "Regex::Parse name = '" << name<< "' ";
      //Regex::Parse(name, &regex_).PermitUncheckedError();
      Status s = Regex::Parse(name, &regex_);
      std::cout << "Status = " << s.ToString() << std::endl;
    }

With DEBUG_LEVEL=2 on the RHEL7 host with g++ 4.8.5

> ./ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_1
Regex::Parse name = 'posix://.*' Status = OK
Regex::Parse name = 'FileChecksumGenCrc32cFactory' Status = OK
Regex::Parse name = 'rocksdb.Noop' Status = OK
Regex::Parse name = 'fixed:[0-9]+' Status = Invalid argument: regex_error
Regex::Parse name = 'rocksdb.FixedPrefix' Status = OK
Regex::Parse name = 'rocksdb.FixedPrefix\.[0-9]+' Status = Invalid argument: regex_error
Regex::Parse name = 'capped:[0-9]+' Status = Invalid argument: regex_error
Regex::Parse name = 'rocksdb.CappedPrefix(\.[0-9]+)?' Status = Invalid argument: regex_error
Regex::Parse name = 'RemoveEmptyValueCompactionFilter' Status = OK
Regex::Parse name = '(VectorRepFactory|vector)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = '(SkipListFactory|skip_list)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = '(HashLinkListRepFactory|hash_linkedlist)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = '(HashSkipListRepFactory|prefix_hash)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = 'cuckoo' Status = OK
ldb: util/regex.cc:29: bool rocksdb::Regex::Matches(const string&) const: Assertion `false' failed.
Aborted

With DEBUG_LEVEL=0 on the RHEL7 host with g++ 4.8.5

> ./ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_1
Regex::Parse name = 'posix://.*' Status = OK
Regex::Parse name = 'FileChecksumGenCrc32cFactory' Status = OK
Regex::Parse name = 'rocksdb.Noop' Status = OK
Regex::Parse name = 'fixed:[0-9]+' Status = Invalid argument: regex_error
Regex::Parse name = 'rocksdb.FixedPrefix' Status = OK
Regex::Parse name = 'rocksdb.FixedPrefix\.[0-9]+' Status = Invalid argument: regex_error
Regex::Parse name = 'capped:[0-9]+' Status = Invalid argument: regex_error
Regex::Parse name = 'rocksdb.CappedPrefix(\.[0-9]+)?' Status = Invalid argument: regex_error
Regex::Parse name = 'RemoveEmptyValueCompactionFilter' Status = OK
Regex::Parse name = '(VectorRepFactory|vector)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = '(SkipListFactory|skip_list)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = '(HashLinkListRepFactory|hash_linkedlist)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = '(HashSkipListRepFactory|prefix_hash)(:[0-9]*)?' Status = Invalid argument: regex_error
Regex::Parse name = 'cuckoo' Status = OK
Regex::Parse name = '(StringAppendOperator|stringappend)' Status = OK
Regex::Parse name = '(StringAppendTESTOperator|stringappendtest)' Status = OK
Regex::Parse name = '(MergeSortOperator|sortlist)' Status = OK
Regex::Parse name = '(BytesXOR|bytesxor)' Status = OK
Regex::Parse name = 'SstPartitionerFixedPrefixFactory' Status = OK
Regex::Parse name = 'BlockBasedTable' Status = OK
Regex::Parse name = 'PlainTable' Status = OK
Regex::Parse name = 'CuckooTable' Status = OK
Regex::Parse name = 'FlushBlockBySizePolicyFactory' Status = OK
Regex::Parse name = 'FlushBlockEveryKeyPolicyFactory' Status = OK
Regex::Parse name = 'leveldb.BytewiseComparator' Status = OK
Regex::Parse name = 'rocksdb.ReverseBytewiseComparator' Status = OK
OK

With g++ version 5.3.1 on the same RHEL7 host

> CC=/opt/rh/devtoolset-4/root/usr/bin/cc CXX=/opt/rh/devtoolset-4/root/usr/bin/g++ make -j12 LIB_MODE=shared DEBUG_LEVEL=2 ldb
> ./ldb put key value --create_if_missing --db=/tmp/test_rocksdb_6_26_1
Regex::Parse name = 'posix://.*' Status = OK
Regex::Parse name = 'FileChecksumGenCrc32cFactory' Status = OK
Regex::Parse name = 'rocksdb.Noop' Status = OK
Regex::Parse name = 'fixed:[0-9]+' Status = OK
Regex::Parse name = 'rocksdb.FixedPrefix' Status = OK
Regex::Parse name = 'rocksdb.FixedPrefix\.[0-9]+' Status = OK
Regex::Parse name = 'capped:[0-9]+' Status = OK
Regex::Parse name = 'rocksdb.CappedPrefix(\.[0-9]+)?' Status = OK
Regex::Parse name = 'RemoveEmptyValueCompactionFilter' Status = OK
Regex::Parse name = '(VectorRepFactory|vector)(:[0-9]*)?' Status = OK
Regex::Parse name = '(SkipListFactory|skip_list)(:[0-9]*)?' Status = OK
Regex::Parse name = '(HashLinkListRepFactory|hash_linkedlist)(:[0-9]*)?' Status = OK
Regex::Parse name = '(HashSkipListRepFactory|prefix_hash)(:[0-9]*)?' Status = OK
Regex::Parse name = 'cuckoo' Status = OK
Regex::Parse name = '(StringAppendOperator|stringappend)' Status = OK
Regex::Parse name = '(StringAppendTESTOperator|stringappendtest)' Status = OK
Regex::Parse name = '(MergeSortOperator|sortlist)' Status = OK
Regex::Parse name = '(BytesXOR|bytesxor)' Status = OK
Regex::Parse name = 'SstPartitionerFixedPrefixFactory' Status = OK
Regex::Parse name = 'BlockBasedTable' Status = OK
Regex::Parse name = 'PlainTable' Status = OK
Regex::Parse name = 'CuckooTable' Status = OK
Regex::Parse name = 'FlushBlockBySizePolicyFactory' Status = OK
Regex::Parse name = 'FlushBlockEveryKeyPolicyFactory' Status = OK
Regex::Parse name = 'leveldb.BytewiseComparator' Status = OK
Regex::Parse name = 'rocksdb.ReverseBytewiseComparator' Status = OK
OK
pdillinger commented 2 years ago

If we revert #9264 for now, @ajkr suggested a possible fix along the lines of replacing optional matching parts by ".*", to effectively get prefix matching. This wouldn't work well if someone registers "foo.*" and someone else "foobar.*" (though it would if instead of using regexes, "foo" and "foobar" are registered prefixes and we always select the longest match).

Perhaps the "more than a prefix name" level of detail is intended to protect from cases like fixed in #9343 . @mrambacher thoughts?

ajkr commented 2 years ago

Perhaps the "more than a prefix name" level of detail is intended to protect from cases like fixed in #9343

I think #9264 doesn't protect against the case of #9343. That's because the non-test use cases of AddNumber() set optional == true beforehand, so the separator/number can still be omitted just the same.

It helps prevent matching invalid strings like "rocksdb.CappedPrefix.a", though OTOH I am not sure the reason to expand our support for parameters in the ID string. Isn't this the old way of configuration?

mrambacher commented 2 years ago

In #9358, I changed it so that "rocksdb.CappedPrefix" is not valid and will not match.