search

prysmaticlabs / prysm

Go implementation of Ethereum proof of stake
https://www.offchainlabs.com/
GNU General Public License v3.0
3.4k stars 954 forks source link

Beacon node stops syncing because >64998 subdirectories in blobs folder #13880

Open Belkaar opened 2 months ago

Belkaar commented 2 months ago

Describe the bug

The Beacon stops syncing at slot 8854183. It has created 64999 directories in the blobs folder and the fs (ext4 in this case) does not allow any more (without setting dir_nlink feature)

A workaround is setting dir_nlink on the ext4 fs: tune2fs -O dir_nlink

Maybe the client should try to avoid this by using a deeper directory structure to store hashes?

Has this worked before in a previous version?

Unknown

🔬 Minimal Reproduction

1) Put data dir on normally formated ext4 fs 2) Try to sync beacon chain

Error

Apr 14 10:06:44 eth-consensus beacon-chain_active[72397]: time="2024-04-14 10:06:44" level=warning msg="Skip processing batched blocks" error="could not process block in batch: could not validate blob data availability at slot 8854183: failed to save BlobSidecar index 0 for block 0xe546bfeee9cdfb74664dafded853a707e342b40691863eea6929a4f1eef61fa3: mkdir /mnt/store_0/beacon/blobs/0xe546bfeee9cdfb74664dafded853a707e342b40691863eea6929a4f1eef61fa3: too many links" prefix=initial-sync

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

5.0.3

Anything else relevant (validator index / public key)?

No response

Belkaar commented 2 months ago

The workaround and the limitation can be found here: https://man7.org/linux/man-pages/man5/ext4.5.html