Closed 0xDmtri closed 2 weeks ago
can confirm that the issue persists with 1.0.3 too.
can also confirm that I have the latest libc6.
ldd (Debian GLIBC 2.36-9+deb12u7) 2.36
can you also reproduce this without RUSTFLAGS="-C target-cpu=native"
?
The same segmentation error is present on our machine as well.
The characteristics are the same except:
Sepolia
Full (No special pruning: --full)
The same segmentation fault was also present when compiiled with the regular --release flag as a second attempt RUSTFLAGS="-C target-cpu=native" cargo build --profile release --features jemalloc,asm-keccak
can you also reproduce this without
RUSTFLAGS="-C target-cpu=native"
?
Ok I can confirm that it works fine without it. Anything else I shall provide?
@mattsse i am happy to do investigation/fixing if you could just point me to the right direction ser :)
@0xDmtri can you run with target-cpu=native
, but without --profile maxperf
? It should display a more detailed backtrace.
@0xDmtri can you run with
target-cpu=native
, but without--profile maxperf
? It should display a more detailed backtrace.
@shekhirin Lol its even weirder now, with debug profile it just works. But I also tried with release profile and it crashed with the same trace back as above. Must be some optimizations?
@0xDmtri can you try with --profile profiling
?
@shekhirin really nothing informative, i used profiling profile and set log lvl as trace and heres what i got:
2024-07-24T01:04:03.081702Z TRACE try_insert_new_payload:try_insert_validated_block{block=BlockNumHash { number: 20372987, hash: 0x7350a601616c6f88d48898d3908fd8afbce9d540abf400dde5b8b0a4234cb5db }}:try_append_canonical_chain: trie::hash_builder: updating merkle tree current=Nibbles("06040e000d") succeeding=Nibbles("06040e000e")
2024-07-24T01:04:03.081780Z TRACE try_insert_new_payload:try_insert_validated_block{block=BlockNumHash { number: 20372987, hash: 0x7350a601616c6f88d48898d3908fd8afbce9d540abf400dde5b8b0a4234cb5db }}:try_append_canonical_chain:loop{i=0 current=Nibbles("06040e000d") build_extensions=false}: trie::hash_builder: prefix lengths after comparing keys len=4 common_prefix_len=4 preceding_len=4 preceding_exists=true
2024-07-24T01:04:03.081786Z TRACE try_insert_new_payload:try_insert_validated_block{block=BlockNumHash { number: 20372987, hash: 0x7350a601616c6f88d48898d3908fd8afbce9d540abf400dde5b8b0a4234cb5db }}:try_append_canonical_chain:loop{i=0 current=Nibbles("06040e000d") build_extensions=false}: trie::hash_builder: extra_digit=13 groups=[TrieMask(0000000000111111), TrieMask(0000000000001111), TrieMask(0011111111111111), TrieMask(0000000000000000), TrieMask(0011111111111111)]
2024-07-24T01:04:03.081791Z TRACE try_insert_new_payload:try_insert_validated_block{block=BlockNumHash { number: 20372987, hash: 0x7350a601616c6f88d48898d3908fd8afbce9d540abf400dde5b8b0a4234cb5db }}:try_append_canonical_chain:loop{i=0 current=Nibbles("06040e000d") build_extensions=false}: trie::hash_builder: skipping 5 nibbles
2024-07-24T01:04:03.081796Z TRACE try_insert_new_payload:try_insert_validated_block{block=BlockNumHash { number: 20372987, hash: 0x7350a601616c6f88d48898d3908fd8afbce9d540abf400dde5b8b0a4234cb5db }}:try_append_canonical_chain:loop{i=0 current=Nibbles("06040e000d") build_extensions=false}: trie::hash_builder: short_node_key=Nibbles("")
2024-07-24T01:04:03.081799Z TRACE try_insert_new_payload:try_insert_validated_block{block=BlockNumHash { number: 20372987, hash: 0x7350a601616c6f88d48898d3908fd8afbce9d540abf400dde5b8b0a4234cb5db }}:try_append_canonical_chain:loop{i=0 current=Nibbles("06040e000d") build_extensions=false}: trie::hash_builder: pushing branch node hash hash=0xfd13804cb34d696e99841d2b47d38be87aef489f644667fa8ba2c681b01ac683Segmentation fault
@shekhirin Ok its even more confusing now.
I ran Valgrind (just in case) - all good no error.
Then, I ran maxperf binary (with debug not stripped) - same crash, same non informative error.
Finally, I ran it via GDB and its not crashing (its been an hour since its running). Hello Mr. Heisenbug :)
I don't know what going on and what else I can do to figure it out.
P.S. Also tried compiling with 1.80, it still crash almost instantly (without the GDB).
This issue is stale because it has been open for 21 days with no activity.
This issue was closed because it has been inactive for 7 days since being marked as stale.
Describe the bug
Node does not start and crashes with segmentation error.
Steps to reproduce
Node logs