search

OffchainLabs / nitro

Nitro goes vroom and fixes everything
Other
749 stars 444 forks source link

Arbitrum Full Node Ledger Growing Rapidly - Possible Bloat Issue #2615

Open darinvhs opened 2 months ago

darinvhs commented 2 months ago

Description I've observed that the Arbitrum Nitro full node ledger is growing unusually fast. On one node, the ledger has reached 5TB, while on another node, it has expanded to 8TB. This rapid and excessive growth seems abnormal and might indicate a memory leak or bloated data issue.

Steps to Reproduce

Context Running du -hsx * in the data directory shows the following disk usage:

du -hs /var/arbitrum-fullnode
5.2T    /var/arbitrum-fullnode

du -hsx *
5.0T    l2chaindata

Setup Details

   spec:
      containers:
      - args:
        - --chain.id=42161
        - --parent-chain.blob-client.beacon-url=censored
        - --parent-chain.connection.url=censored
        - --node.feed.input.url=wss://arb1.arbitrum.io/feed
        - --http.addr=0.0.0.0
        - --http.api=eth,net,web3,debug,txpool
        - --http.vhosts=*
        - --http.corsdomain=null
        - --ws.addr=0.0.0.0
        - --ws.api=eth,net,web3,debug,txpool
        - --ws.origins=*
        - --execution.rpc.tx-fee-cap=100
        - --execution.rpc.gas-cap=500000000
        - --metrics
        - --metrics-server.addr=0.0.0.0
        - --metrics-server.port=6070
        image: offchainlabs/nitro-node:v3.1.0-7d1d84c

Impact This could lead to very high storage requirements, making it difficult to maintain full nodes. It might be helpful to investigate to prevent potential issues with disk space or performance.

ThomasBlock commented 2 months ago

38 TB here after one year. i will just reinstall

only hint i got so far is to make sure that the process exits gracefully.

0x090909 commented 1 month ago

I agree, currently the system only allows to prune the state, while the ancient blocks are still present.

By also adding ancient data prune we can save up around 0.5 TB at the current sate of sync.

Plus by pruning the state we can lower the whole database size to around 600 gb.

I did open already an issue on the ancient pruning but nobody is looking into it

2585