westerndigitalcorporation / zenfs

ZenFS is a storage backend for RocksDB that enables support for ZNS SSDs and SMR HDDs.
GNU General Public License v2.0
239 stars 87 forks source link

Received signal 11 (Segmentation fault) arise when using db_bench testing zenfs #140

Closed ZhuWeiLin0 closed 2 years ago

ZhuWeiLin0 commented 2 years ago

hello, everybody, I'm a newbie to zenfs and rocksdb, so any help is deeply appreciated!!

segmentation fault arise when I try to test zenfs with db_bench I used the newest version of rocksdb and zenfs, and i tried to solve this problem by setting "ignore_unknown_options=true" in ConfigOptions(every ignore_unknown_options is set to true in rocksdb/include/rocksdb/convenience.h), according to rocksdb#9333 . I also tried the solution mentioned in #126 , however, nothing worked. i notice that #115 have the same problem

# ./db_bench --fs_uri=zenfs://nullb2 --benchmarks=fillrandom --use_direct_io_for_flush_and_compaction
Received signal 11 (Segmentation fault)
#0   ./db_bench(+0x2a1f28) [0x563a7734bf28] ??  ??:0    
#1   ./db_bench(+0xb6e64) [0x563a77160e64] ??   ??:0    
#2   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f98ee7330b3] ??    ??:0    
#3   ./db_bench(+0xb587e) [0x563a7715f87e] ??   ??:0    
segmentation fault (core dump)

this is really driving me crazy. again, any help is deeply appreciated!!!!! thanks!!!!

skyzh commented 2 years ago

Can you first compile RocksDB in debug mode and print the full stack trace?

skyzh commented 2 years ago

Also, IIRC, nullb2 seems like a nullblk device, which is not well-supported by libzbd.

aravind-wdc commented 2 years ago

@ZhuWeiLin0 Could you try with --fs_uri=zenfs://dev:nullb2 (assuming /dev/nullb2 is an emulated zoned device.)

damien-lemoal commented 2 years ago

Also, IIRC, nullb2 seems like a nullblk device, which is not well-supported by libzbd.

That is not true at all. The behavior of libzbd on nullblk is as good as the device zone management behavior, which depends on how the nullblk device is configured, and als depends on the kernel version (not all features are supported by all kernels).

ZhuWeiLin0 commented 2 years ago

@ZhuWeiLin0 Could you try with --fs_uri=zenfs://dev:nullb2 (assuming /dev/nullb2 is an emulated zoned device.)

OMG thanks, it worked!!!!!! so i guess the ultimate solution is [ set every ignore_unknown_options to true in rocksdb/include/rocksdb/convenience.h and never forget the dev prefix

ZhuWeiLin0 commented 2 years ago

Also, IIRC, nullb2 seems like a nullblk device, which is not well-supported by libzbd.

thanks a lot for your reply! i add 'dev' before 'nullb2' and it worked : )

aravind-wdc commented 2 years ago

@ZhuWeiLin0 Could you try with --fs_uri=zenfs://dev:nullb2 (assuming /dev/nullb2 is an emulated zoned device.)

OMG thanks, it worked!!!!!! so i guess the ultimate solution is [ set every ignore_unknown_options to true in rocksdb/include/rocksdb/convenience.h and never forget the dev prefix

No need to set the "--ignore_unknown_options". The "--fs_uri=..." string needs to be in correct format, then it all works. When it is not in expected format, zenfs returns error, which is not passed cleanly by rocksdb to db_bench, which is causing this crash. All the other issues that you mentioned are tracking this and trying to fix it so even if a wrong input is given, the application fails instead of crashing. Thanks.

ZhuWeiLin0 commented 2 years ago

@ZhuWeiLin0 Could you try with --fs_uri=zenfs://dev:nullb2 (assuming /dev/nullb2 is an emulated zoned device.)

OMG thanks, it worked!!!!!! so i guess the ultimate solution is [ set every ignore_unknown_options to true in rocksdb/include/rocksdb/convenience.h and never forget the dev prefix

No need to set the "--ignore_unknown_options". The "--fs_uri=..." string needs to be in correct format, then it all works. When it is not in expected format, zenfs returns error, which is not passed cleanly by rocksdb to db_bench, which is causing this crash. All the other issues that you mentioned are tracking this and trying to fix it so even if a wrong input is given, the application fails instead of crashing. Thanks.

but i'm pretty sure i tried the correct format without setting the "ignore_unknown_options", it failed

aravind-wdc commented 2 years ago

but i'm pretty sure i tried the correct format without setting the "ignore_unknown_options", it failed

Could you please confirm it ?