Bug: Segmentation Fault Error at indexing blocks [180001..182000]

adriankae commented 4 months ago

Describe the bug Everytime I start electrs I get a seg fault error at indexing the blocks. Specifically at blocks [180001..182000]. It's always in the same spot. Any help would be appreciated.

Electrs version v.0.10.5

To Reproduce Steps to reproduce the behavior:

Configure and start electrs
Wait
See error

Expected behavior Indexing throws no errors.

Configuration

electrs.toml

Didn't find it in the directory.

[2024-07-12T16:42:00.610Z INFO  electrs::index] indexing 2000 blocks: [180001..182000]
Segmentation fault (core dumped)

running with gdb:


Thread 8 "rocksdb:high" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff58006c0 (LWP 3120)]
0x0000555555ba5775 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()

Environment variables: ELECTRS_X=Y;... Arguments: --foo

System running electrs

Deployment method: manual from source
OS name and version: Ubuntu Server 24.04 LTS

Electrum client Client name (if not upstream desktop Electrum) and version: no electrum installed

Additional context n/a

adriankae commented 4 months ago

error running with gdb:

[2024-07-12T16:55:11.385Z INFO electrs::index] indexing 2000 blocks: [180001..182000]

Thread 8 "rocksdb:high" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff58006c0 (LWP 3120)] 0x0000555555ba5775 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()

Kixunil commented 4 months ago

Did you link dynamically or statically?

adriankae commented 4 months ago

Did you link dynamically or statically?

I linked statically.

Kixunil commented 4 months ago

That's really weird, probably has something to do with C++ compilation. Makes me rethink my previous claim that maybe pure Rust impl isn't that interesting.

adriankae commented 4 months ago

When I try to link dynamically i get these errors:

error[E0425]: cannot find function `rocksdb_block_based_options_set_block_cache_compressed` in crate `ffi`
   --> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:532:18
    |
532 |             ffi::rocksdb_block_based_options_set_block_cache_compressed(self.inner, cache);
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `rocksdb_block_based_options_set_block_cache`
    |
   ::: /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:59227
    |
3   | ... { pub fn rocksdb_block_based_options_set_block_cache (options : * mut rocksdb_block_based_table_options_t , block_cache : * mut rocksdb_cache_t) ; ...
    |       ---------------------------------------------------------------------------------------------------------------------------------------------- similarly named function `rocksdb_block_based_options_set_block_cache` defined here

error[E0425]: cannot find function `rocksdb_block_based_options_set_block_cache_compressed` in crate `ffi`
   --> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:553:18
    |
553 |             ffi::rocksdb_block_based_options_set_block_cache_compressed(self.inner, cache.0.inner);
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `rocksdb_block_based_options_set_block_cache`
    |
   ::: /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:59227
    |
3   | ... { pub fn rocksdb_block_based_options_set_block_cache (options : * mut rocksdb_block_based_table_options_t , block_cache : * mut rocksdb_cache_t) ; ...
    |       ---------------------------------------------------------------------------------------------------------------------------------------------- similarly named function `rocksdb_block_based_options_set_block_cache` defined here

error[E0061]: this function takes 9 arguments but 5 arguments were supplied
    --> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:2525:13
     |
2525 |               ffi::rocksdb_options_set_plain_table_factory(
     |  _____________^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^-
2526 | |                 self.inner,
2527 | |                 options.user_key_length,
2528 | |                 options.bloom_bits_per_key,
2529 | |                 options.hash_table_ratio,
2530 | |                 options.index_sparseness,
2531 | |             );
     | |_____________- multiple arguments are missing
     |
note: function defined here
    --> /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:88313
     |
3    | ...size) ; } extern "C" { pub fn rocksdb_options_set_plain_table_factory (arg1 : * mut rocksdb_options...
     |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: provide the arguments
     |
2525 |             ffi::rocksdb_options_set_plain_table_factory(self.inner, options.user_key_length, options.bloom_bits_per_key, options.hash_table_ratio, options.index_sparseness, /* usize */, /* i8 */, /* u8 */, /* u8 */);
     |                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some errors have detailed explanations: E0061, E0425.
For more information about an error, try `rustc --explain E0061`.
error: could not compile `electrs-rocksdb` (lib) due to 3 previous errors

Kixunil commented 4 months ago

Yes, Ubuntu has a different rocksdb version than Debian. I wish they didn't break the API so much.

adriankae commented 4 months ago

Is there anything I could do? I'm pretty much out of ideas...

Kixunil commented 4 months ago

I don't have many ideas either. Maybe try switching compiler to clang? (If you used g++, or vice-versa).

romanz commented 3 months ago

@adriankae Does the error reproduce if the index DB is rebuilt from scratch?

bkjaerland commented 3 months ago

I have the exact same issue. For me it did not help to rebuild the DB from scratch.

bkjaerland commented 3 months ago

Found the issue for me, I was missing the build-essential package. After installing it and rebuilding electrs it works like a charm.

Kixunil commented 3 months ago

How come it even compiled without it?! That's super weird.

antonilol commented 2 months ago

Is this a rocksdb issue? I think somewhere a compile time error should be introduced.

xjmzx commented 2 months ago

Confirming the same issue (at same block height) and the same fix build-essential was not installed.

OS: Ubuntu 24.04.1 LTS x86_64 Kernel: 6.8.0-41-generic

jozbone commented 2 months ago

Just confirming as well that I was seeing the same issue at the same block height on Ubuntu 24.04. Installed "build-essential" and rebuilt electrs (static linking). After that I was able to run it and get past the previously failing block height point.

Kixunil commented 2 months ago

I don't think we need more reports, but new people coming to the issue can just thumbs up this comment: https://github.com/romanz/electrs/issues/1061#issuecomment-2308759609

What would be really helpful is to abort compilation if this is happening but that requires some serious digging and will likely need an issue filed against the cc crate (but only once it's confirmed to belong there - please don't pester them about it now).

If someone figures out more information about this that would explain why this happens that'd be great. Sadly I don't have that much time for this kind of thing lately.

romanz commented 1 month ago

On Ubuntu 24.04 installing clang results in Clang 18:

$ c++ --version
Ubuntu clang version 18.1.3 (1ubuntu1)
...

If I then install build-essential, GCC 13 becomes the default:

$ c++ --version
c++ (Ubuntu 13.2.0-23ubuntu4) 13.2.0
...

Also, https://github.com/mempool/electrs/pull/106 also stopped using Clang 18.

romanz commented 3 weeks ago

Reproduced using a Ubuntu 24.04 Docker container:

$ gdb --args /usr/bin/electrs --log-filters INFO
(gdb) r
...
[2024-10-29T14:15:03.607Z INFO  electrs::chain] chain updated: tip=00000000000004ff83b6c10460b239ef4a6aa320e5fffd6c7bcedefa8c78593c, height=180000
[2024-10-29T14:15:03.608Z INFO  electrs::index] indexing 2000 blocks: [180001..182000]

Thread 10 "rocksdb:high" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7bb3c1e006c0 (LWP 1032)]
0x000064b8f5f80795 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
(gdb) bt
#0  0x000064b8f5f80795 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
#1  0x000064b8f5f8074c in rocksdb::CompressionContextCache::GetCachedZSTDUncompressData() ()
#2  0x000064b8f5f177c7 in rocksdb::UncompressionContext::UncompressionContext(rocksdb::CompressionType) ()
#3  0x000064b8f5f4c054 in rocksdb::BlockFetcher::ReadBlockContents() ()
#4  0x000064b8f5f1f413 in rocksdb::Status rocksdb::(anonymous namespace)::ReadBlockFromFile<rocksdb::Block>(rocksdb::RandomAccessFileReader*, rocksdb::FilePrefetchBuffer*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, std::unique_ptr<rocksdb::Block, std::default_delete<rocksdb::Block> >*, rocksdb::ImmutableOptions const&, bool, bool, rocksdb::BlockType, rocksdb::UncompressionDict const&, rocksdb::PersistentCacheOptions const&, unsigned long, rocksdb::MemoryAllocator*, bool, bool, rocksdb::FilterPolicy const*, bool) ()
#5  0x000064b8f5f164a5 in rocksdb::Status rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::Block>(rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::UncompressionDict const&, rocksdb::CachableEntry<rocksdb::Block>*, rocksdb::BlockType, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, bool, bool, bool, bool) const ()
#6  0x000064b8f5f431f3 in rocksdb::BlockBasedTable::IndexReaderCommon::ReadIndexBlock(rocksdb::BlockBasedTable const*, rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocksdb::CachableEntry<rocksdb::Block>*) ()
#7  0x000064b8f6035b8e in rocksdb::BinarySearchIndexReader::Create(rocksdb::BlockBasedTable const*, rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader> >*) ()
#8  0x000064b8f5f25d54 in rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader> >*) ()
#9  0x000064b8f5f250b6 in rocksdb::BlockBasedTable::PrefetchIndexAndFilterBlocks(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, rocksdb::BlockBasedTable*, bool, rocksdb::BlockBasedTableOptions const&, int, unsigned long, unsigned long, rocksdb::BlockCacheLookupContext*) ()
#10 0x000064b8f5f22706 in rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::CacheReservationManager>, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats*, rocksdb::BlockCacheTracer*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::array<unsigned long, 2ul>) ()
#11 0x000064b8f5f0ca46 in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool) const ()
#12 0x000064b8f5e2c61c in rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, bool, bool, rocksdb::HistogramImpl*, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, int, bool, unsigned long, rocksdb::Temperature) ()
#13 0x000064b8f5e2b497 in rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Cache::Handle**, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature) ()
#14 0x000064b8f5e2cb10 in rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::RangeDelAggregator*, std::shared_ptr<rocksdb::SliceTransform const> const&, rocksdb::TableReader**, rocksdb::HistogramImpl*, rocksdb::TableReaderCaller, rocksdb::Arena*, bool, int, unsigned long, rocksdb::InternalKey const*, rocksdb::InternalKey const*, bool, rocksdb::TruncatedRangeDelIterator**) ()
#15 0x000064b8f5fbc8a8 in rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*) ()
#16 0x000064b8f5de44ab in rocksdb::FlushJob::WriteLevel0Table() ()
#17 0x000064b8f5de040d in rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*, bool*) ()
#18 0x000064b8f5d71fe7 in rocksdb::DBImpl::FlushMemTableToOutputFile(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, bool*, rocksdb::JobContext*, rocksdb::SuperVersionContext*, std::vector<unsigned long, std::allocator<unsigned long> >&, unsigned long, rocksdb::SnapshotChecker*, rocksdb::LogBuffer*, rocksdb::Env::Priority) ()
#19 0x000064b8f5d73238 in rocksdb::DBImpl::FlushMemTablesToOutputFiles(rocksdb::autovector<rocksdb::DBImpl::BGFlushArg, 8ul> const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) ()
#20 0x000064b8f5d85428 in rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::FlushReason*, rocksdb::Env::Priority) ()
#21 0x000064b8f5d83b88 in rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) ()
#22 0x000064b8f5f8e4f5 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#23 0x000064b8f5f8e759 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#24 0x00007bb3c488fbb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#25 0x00007bb3c4519a94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#26 0x00007bb3c45a6a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

with the following diff:

diff --git a/Dockerfile b/Dockerfile
index c8ec506e..a6f1dba8 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -2,19 +2,18 @@
 # The maintainers of electrs are not deeply familiar with Docker, so you should DYOR.
 # If you are not familiar with Docker either it's probably be safer to NOT use it.

-FROM debian:bookworm-slim AS base
+FROM ubuntu:24.04 AS base
 RUN apt-get update -qqy
-RUN apt-get install -qqy librocksdb-dev curl
+RUN apt-get install -qqy curl gdb bash

 ### Electrum Rust Server ###
 FROM base AS electrs-build
RUN apt-get install -qqy cargo clang cmake

 # Install electrs
 WORKDIR /build/electrs
 COPY . .
-ENV ROCKSDB_INCLUDE_DIR=/usr/include
-ENV ROCKSDB_LIB_DIR=/usr/lib
 RUN cargo install --locked --path .

 FROM base AS result

schmitmd commented 6 days ago

I can confirm that on FreeBSD, installing llvm17 and recompiling electrs with the following set of variables does indeed fix this.

CC=/usr/local/bin/clang17
CXX=/usr/local/bin/clang++17
CPP=/usr/local/bin/clang-cpp17
RUSTFLAGS="-C linker=clang17"

romanz / electrs

Bug: Segmentation Fault Error at indexing blocks [180001..182000] #1061