Node got stuck on stage [3/16 Batches] Waiting for at least one new block

[andreclaro]

System information

[cdk-erigon-lib] timestamp 2024-03-12:16:34 cdk-erigon version 2.43.0-dev-3e3013f2

OS & Version: Windows/Linux/OSX Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy

Erigon Command (with flags/config):

/usr/local/bin/cdk-erigon \
    --datadir /var/lib/xlayer/data/erigon \
    --chain xlayer-testnet \
    --private.api.addr localhost:9091 \
    --zkevm.l2-chain-id 195 \
    --zkevm.l2-sequencer-rpc-url https://testrpc.xlayer.tech \
    --zkevm.l2-datastreamer-url teststream.xlayer.tech:8800 \
    --zkevm.l1-chain-id 11155111 \
    --zkevm.l1-rpc-url http://ETH-SEPOLIA_ADDRESS:8545 \
    --zkevm.address-sequencer "" \
    --zkevm.address-zkevm "" \
    --zkevm.address-admin "0x6662621411A8DACC3cA7049C8BddABaa9a999ce3" \
    --zkevm.address-rollup "0x01469dACfDDA885D68Ff0f8628F2629c14F95a20" \
    --zkevm.address-ger-manager "0x66E61bA00F58b857A9DD2C500F3aBc424A46BD20" \
    --zkevm.l1-rollup-id 1 \
    --zkevm.l1-first-block 4648290 \
    --zkevm.l1-block-range 100 \
    --zkevm.l1-query-delay 6000 \
    --zkevm.rpc-ratelimit 250 \
    --zkevm.datastream-version 2 \
    --externalcl \
    --http \
    --http.api "eth,debug,net,trace,web3,erigon,zkevm" \
    --http.addr 0.0.0.0 \
    --http.port 8545 \
    --http.vhosts "*" \
    --ws \
    --metrics \
    --metrics.addr 0.0.0.0 \
    --metrics.port 6060 \
    --authrpc.addr 127.0.0.1 \
    --authrpc.port 8551 \
    --authrpc.jwtsecret /var/lib/xlayer/jwt.hex

Chain/Network: X Layer Testnet

Expected behaviour

No client restart required to fix the issue.

Actual behaviour

Node got stuck without processing new blocks on the stage 3/16 Batches with the logging message showing only: [3/16 Batches] Waiting for at least one new block.

Client was restarted and the node started to process new blocks again.

Steps to reproduce the behaviour

Don't know

Logs

cdk-erigon[740162]: [INFO] [05-30|14:49:32.569] [3/16 Batches] Waiting for at least one new block.
cdk-erigon[740162]: [INFO] [05-30|14:49:42.569] [3/16 Batches] Waiting for at least one new block.
cdk-erigon[740162]: [INFO] [05-30|14:49:52.569] [3/16 Batches] Waiting for at least one new block.
cdk-erigon[740162]: [INFO] [05-30|14:50:02.570] [3/16 Batches] Waiting for at least one new block.
cdk-erigon[740162]: [INFO] [05-30|14:50:12.570] [3/16 Batches] Waiting for at least one new block.
cdk-erigon[740162]: [INFO] [05-30|14:50:22.570] [3/16 Batches] Waiting for at least one new block

[mandrigin]

Looks like a datastream connection issue, please monitor and report if you see it again.

[mandrigin]

Yep, it seems that sometimes I can see it on our integration networks. Adding to a correct milestone.

[mandrigin]

Moved it to a later milestone, not easy to reproduce, so just not in the blockers for now.

[Sharonbc01]

@mandrigin do we need @ToniRamirezM to help with this issue?

[mandrigin]

seems to be resolved with latest refactorings