Fuzzing rocksdb doesn't work

[agroce]

Another DeepState example: https://github.com/agroce/testleveldb

1. Install DeepState, leveldb, and rocksdb (no need to follow the instructions to add sanitizers or anything, just compile release versions)
2. Change the makefile to compile with a recent clang
3. Try using Eclipser via deepstate-eclipser

Eclipser claims to find a crashing input for the DiffTestDBs executable immediately, but it seems to be just a set of zero bytes that don't actually cause a crash. Incompatibility with something one of the DBs is doing and QEMU?

Fuzzing just leveldb works fine, so it's something RocksDB does. Not high priority, just FYI of an important fuzzing target that seems to break Eclipser.

[jchoi2022]

Thank you for reporting the issue. I will take a look and figure out the cause as soon as possible.

[jchoi2022]

I finally had some time to look into this issue, but setting up environment to reproduce this issue is a little bit tough. I modified the Dockerfile from deepstate repository, and used the attached Dockerfile to build testleveldb repository.

Dockerfile.txt

However, when I run TestLevelDB and DiffTestDBs, they abort immediately as they fail to initialize the database.

user@d1791686ef18:~/testleveldb$ ./TestLevelDB 
TRACE: Running: LevelDB_Fuzz from TestLevelDB.cpp(19)
CRITICAL: TestLevelDB.cpp(27): Could not create the leveldb test database!
ERROR: Failed: LevelDB_Fuzz
user@d1791686ef18:~/testleveldb$ ./DiffTestDBs 
TRACE: Running: LevelDB_Fuzz from DiffTestDBs.cpp(39)
CRITICAL: DiffTestDBs.cpp(47): Could not create the leveldb test database!
ERROR: Failed: LevelDB_Fuzz

Would you take a look at my Dockerfile to identify what's going wrong, or provide a working Dockerfile for testleveldb repository?

[agroce]

@jchoi2022 I suspect it's just that the Makefile sets the location of the databases somewhere that doesn't exist. Edit those to paths that exist and it likely works.

[agroce]

Try with the new paths which are just dirs in the local directory

[agroce]

Oops not the Makefile, Common.hpp. See the changes I just made.

[agroce]

I think after this your dockerfile will work

[jchoi2022]

Thank you for the help, and I could successfully reproduce the issue. Unfortunately, it turned out that this was another QEMU issue, similar to what we witnessed in ASAN (#5). When trying to run DiffTestDBs with pure QEMU (i.e. no additional instrumentation for Eclipser), it crashes while initializing globally declared RocksDB entities.

(gdb) c
Continuing.

Program received signal SIGILL, Illegal instruction.
rocksdb::LRUCache::LRUCache (this=0x9bda00, capacity=8388608, num_shard_bits=4, strict_capacity_limit=false, high_pri_pool_ratio=0, allocator=..., use_adaptive_mutex=false) at cache/lru_cache.cc:471
471   num_shards_ = 1 << num_shard_bits;
(gdb) where
#0  rocksdb::LRUCache::LRUCache (this=0x9bda00, capacity=8388608, num_shard_bits=4, strict_capacity_limit=false, high_pri_pool_ratio=0, allocator=..., use_adaptive_mutex=false) at cache/lru_cache.cc:471
#1  0x00000000005c4084 in __gnu_cxx::new_allocator<rocksdb::LRUCache>::construct<rocksdb::LRUCache, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (
    this=<optimized out>, __p=0x9bda00) at /usr/include/c++/7/ext/new_allocator.h:136
#2  std::allocator_traits<std::allocator<rocksdb::LRUCache> >::construct<rocksdb::LRUCache, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (__a=..., __p=0x9bda00)
    at /usr/include/c++/7/bits/alloc_traits.h:475
#3  std::__shared_ptr<rocksdb::LRUCache, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rocksdb::LRUCache>, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (
    __a=..., __tag=..., this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr_base.h:1327
#4  std::shared_ptr<rocksdb::LRUCache>::shared_ptr<std::allocator<rocksdb::LRUCache>, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (__a=..., __tag=..., 
    this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr.h:344
#5  std::allocate_shared<rocksdb::LRUCache, std::allocator<rocksdb::LRUCache>, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (__a=...)
    at /usr/include/c++/7/bits/shared_ptr.h:691
#6  std::make_shared<rocksdb::LRUCache, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> () at /usr/include/c++/7/bits/shared_ptr.h:707
#7  rocksdb::NewLRUCache (capacity=capacity@entry=8388608, num_shard_bits=4, num_shard_bits@entry=-1, strict_capacity_limit=strict_capacity_limit@entry=false, high_pri_pool_ratio=high_pri_pool_ratio@entry=0, 
    memory_allocator=std::shared_ptr<rocksdb::MemoryAllocator> (empty) = {...}, use_adaptive_mutex=use_adaptive_mutex@entry=false) at cache/lru_cache.cc:566
#8  0x00000000005532ab in rocksdb::BlockBasedTableFactory::BlockBasedTableFactory (this=0x9bd7d0, _table_options=...) at table/block_based/block_based_table_factory.cc:175
#9  0x00000000005378df in rocksdb::ColumnFamilyOptions::ColumnFamilyOptions (this=0x9806c0 <rocksdb::OptionsHelper::dummy_cf_options>) at options/options.cc:103
#10 0x0000000000412aae in __static_initialization_and_destruction_0 (__priority=65535, __initialize_p=1) at options/options_helper.cc:1731
#11 0x00000000006f9b4d in __libc_csu_init ()
#12 0x000000400228cb28 in __libc_start_main (main=0x6f68d0 <main>, argc=3, argv=0x40008006d8, init=0x6f9b00 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x40008006c8)
    at ../csu/libc-start.c:266
#13 0x000000000042945a in _start ()

I tried several versions of QEMU including 2.3.0 and 3.1.0, but the crash persists. I also tried to debug the crash for a while, but it seems tough to fix an issue in QEMU code.