search

testinprod-io / op-erigon

Optimism implementation on the efficiency frontier
https://op-erigon.testinprod.io
GNU Lesser General Public License v3.0
87 stars 15 forks source link

op-node not starting without erigon responding on auth-rpc #76

Closed czarly closed 1 year ago

czarly commented 1 year ago

in the logs of op-node I get the following error message in a crash loop:

lvl=crit msg="Application failed"
message="failed to determine L2BlockRef of height 105235063, could not get payload: not found"

while op-erigon is waiting for the consensus client before advancing it's blocks.

if I replace op-erigon with op-geth in the same configuration then op-node is starting fine and op-geth syncing.

my op-node has no persistent data so I can not just bootstrap it with op-geth and run it with op-erigon later. I guess I get something wrong.

optimism-archive.yml.txt

czarly commented 1 year ago

the typo in erigion is not the cause. I checked and it was not part of the original problem.

the full error message is

t=2023-06-26T15:22:58+0000 lvl=info msg="Rollup Config"            l2_chain_id=10 l2_network=mainnet l1_chain_id=1 l1_network=mainnet l2_start_time=1,686,068,903 l2_block_hash=0xdbf6a80fef073de06add9b0d14026d6e5a86c85f6d102c36d3d8e9cf89c2afd3 l2_block_number=105,235,063 l1_block_hash=0x438335a20d98863a4c0c97999eb2481921ccd28553eac6f913af7c12aec04108 l1_block_number=17,422,590 regolith_time="@ genesis"
t=2023-06-26T15:22:58+0000 lvl=info msg="loaded new runtime config values!" p2p_seq_address=0xAAAA45d9549EDA09E70937013520214382Ffc4A2
t=2023-06-26T15:22:58+0000 lvl=eror msg="Error initializing the rollup node" err="failed to determine L2BlockRef of height 105235063, could not get payload: not found"
t=2023-06-26T15:22:58+0000 lvl=eror msg="Unable to create the rollup node"   error="failed to determine L2BlockRef of height 105235063, could not get payload: not found"
t=2023-06-26T15:22:58+0000 lvl=crit msg="Application failed"                 message="failed to determine L2BlockRef of height 105235063, could not get payload: not found"
ImTei commented 1 year ago

@czarly Thank you for the report. Can you give me full logs of op-erigon?

czarly commented 1 year ago

@ImTei this is all from the erigon logs:

while digging it up I found my mistake. I had used /home/erigon/.local/share/erigon as mount point for the data volume with the chain data. erigon didn't find it's chain data because it looked in /root/...

[INFO] [06-27|10:35:12.943] logging to file system                   log dir=/root/.local/share/erigon/logs file prefix=erigon log level=info json=false
[INFO] [06-27|10:35:12.952] Build info                               git_branch= git_tag= git_commit=
[INFO] [06-27|10:35:12.952] Starting Erigon on Optimism Mainnet... 
[INFO] [06-27|10:35:12.953] Maximum peer count                       ETH=100 total=100
[INFO] [06-27|10:35:12.953] starting HTTP APIs                       APIs=eth,erigon,engine
[INFO] [06-27|10:35:12.953] torrent verbosity                        level=WRN
[INFO] [06-27|10:35:15.073] Set global gas cap                       cap=50000000
[INFO] [06-27|10:35:15.073] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
[INFO] [06-27|10:35:15.074] Opening Database                         label=chaindata path=/root/.local/share/erigon/chaindata
[INFO] [06-27|10:35:15.074] [db] params: growStep=2GB, mapsSize=7TB, shrinkThreshold=-1, pageSize=4KB, label=chaindata, WriteMap=false, Durable=false, NoReadahead=true,  
[INFO] [06-27|10:35:15.079] Re-Opening DB in exclusive mode to apply migrations 
[INFO] [06-27|10:35:15.080] [db] params: growStep=2GB, mapsSize=7TB, shrinkThreshold=-1, pageSize=4KB, label=chaindata, WriteMap=false, Durable=false, NoReadahead=true,  
[INFO] [06-27|10:35:15.081] Apply migration                          name=db_schema_version5
[INFO] [06-27|10:35:15.081] Applied migration                        name=db_schema_version5
[INFO] [06-27|10:35:15.081] Apply migration                          name=txs_begin_end
[INFO] [06-27|10:35:15.082] Applied migration                        name=txs_begin_end
[INFO] [06-27|10:35:15.082] Apply migration                          name=reset_blocks_4
[INFO] [06-27|10:35:15.082] Applied migration                        name=reset_blocks_4
[INFO] [06-27|10:35:15.082] Updated DB schema to                     version=6.0.0
[INFO] [06-27|10:35:15.082] [db] params: growStep=2GB, mapsSize=7TB, shrinkThreshold=-1, pageSize=4KB, label=chaindata, WriteMap=false, Durable=false, NoReadahead=true,  
[INFO] [06-27|10:35:15.085] Writing custom genesis block             hash=0x2b955e7dd8df0ba8a50ee3e5c79065be9a370b1e52398694001067d40f7d6731
[INFO] [06-27|10:35:15.085] Initialised chain configuration          config="{ChainID: 10, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 0, Muir Glacier: 0, Berlin: 105235063, London: 105235063, Arrow Glacier: 105235063, Gray Glacier: 105235063, Terminal Total Difficulty: 0, Merge Netsplit: 105235063, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Engine: unknown}" genesis=0x2b955e7dd8df0ba8a50ee3e5c79065be9a370b1e52398694001067d40f7d6731
[INFO] [06-27|10:35:15.085] Effective                                prune_flags= snapshot_flags= history.v3=false
[INFO] [06-27|10:35:15.088] Initialising Ethereum protocol           network=10
[INFO] [06-27|10:35:15.088] Disk storage enabled for ethash DAGs     dir=/root/.local/share/erigon/ethash-dags count=2
[INFO] [06-27|10:35:15.094] Starting private RPC server              on=localhost:9090
[INFO] [06-27|10:35:15.094] new subscription to logs established 
[INFO] [06-27|10:35:15.094] rpc filters: subscribing to Erigon events 
[INFO] [06-27|10:35:15.094] New txs subscriber joined 
[INFO] [06-27|10:35:15.095] new subscription to newHeaders established 
[INFO] [06-27|10:35:15.095] Reading JWT secret                       path=/jwtsecret
[INFO] [06-27|10:35:15.096] HTTP endpoint opened for Engine API      url=[::]:8551 ws=true ws.compression=true
[INFO] [06-27|10:35:15.096] HTTP endpoint opened                     url=[::]:8545 ws=true ws.compression=true grpc=false
[INFO] [06-27|10:35:15.099] Started P2P networking                   version=68 self=enode://a3e89da111a9cccf4213d8c6e3c52bb98ec61393a793208e78aa38eedcdc89eea6516490bd1d40de6debbc85d09c1a2b16ff53fdef51473cc9fa3973771523d2@127.0.0.1:0 name=erigon/v0.01.5/linux-amd64/go1.20.2
[INFO] [06-27|10:35:15.099] Started P2P networking                   version=67 self=enode://a3e89da111a9cccf4213d8c6e3c52bb98ec61393a793208e78aa38eedcdc89eea6516490bd1d40de6debbc85d09c1a2b16ff53fdef51473cc9fa3973771523d2@127.0.0.1:0 name=erigon/v0.01.5/linux-amd64/go1.20.2
[INFO] [06-27|10:35:15.104] [txpool] Started 
[INFO] [06-27|10:35:15.105] [2/15 Headers] Waiting for Consensus Layer... 
[INFO] [06-27|10:36:15.095] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=24.4MB sys=54.7MB
dappnodedev commented 1 year ago

Hey @czarly @ImTei ! I am getting the same error:

op-node:

t=2023-09-22T09:51:15+0000 lvl=info msg="Initializing Rollup Node"
t=2023-09-22T09:51:15+0000 lvl=info msg="Rollup Config"            l2_chain_id=10 l2_network=mainnet l1_chain_id=1 l1_network=mainnet l2_start_time=1,686,068,903 l2_block_hash=0xdbf6a80fef073de06add9b0d14026d6e5a86c85f6d102c36d3d8e9cf89c2afd3 l2_block_number=105,235,063 l1_block_hash=0x438335a20d98863a4c0c97999eb2481921ccd28553eac6f913af7c12aec04108 l1_block_number=17,422,590 regolith_time="@ genesis"
t=2023-09-22T09:51:16+0000 lvl=info msg="loaded new runtime config values!" p2p_seq_address=0xAAAA45d9549EDA09E70937013520214382Ffc4A2
t=2023-09-22T09:51:16+0000 lvl=eror msg="Error initializing the rollup node" err="failed to determine L2BlockRef of height 105235063, could not get payload: not found"
t=2023-09-22T09:51:16+0000 lvl=eror msg="Unable to create the rollup node"   error="failed to determine L2BlockRef of height 105235063, could not get payload: not found"
t=2023-09-22T09:51:16+0000 lvl=crit msg="Application failed"                 message="failed to determine L2BlockRef of height 105235063, could not get payload: not found"

op-erigon:

[INFO - entrypoint] Enabling historical RPC
[INFO - entrypoint] Database already exists, skipping initialization
[INFO - entrypoint] Starting Erigon
[INFO] [09-22|09:39:43.706] logging to file system                   log dir=/data/logs file prefix=erigon log level=info json=false
[INFO] [09-22|09:39:43.707] Starting metrics server                  addr=http://0.0.0.0:6060/debug/metrics/prometheus
[INFO] [09-22|09:39:43.707] Build info                               git_branch= git_tag= git_commit=
[INFO] [09-22|09:39:43.707] Starting Erigon on Optimism Mainnet... 
[INFO] [09-22|09:39:43.707] Maximum peer count                       ETH=100 total=100
[INFO] [09-22|09:39:43.708] starting HTTP APIs                       APIs=eth,erigon,engine
[INFO] [09-22|09:39:43.708] torrent verbosity                        level=WRN
[INFO] [09-22|09:39:45.810] Set global gas cap                       cap=50000000
[INFO] [09-22|09:39:45.810] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
[INFO] [09-22|09:39:45.810] Opening Database                         label=chaindata path=/data/chaindata
[INFO] [09-22|09:39:45.812] Initialised chain configuration          config="{ChainID: 10, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 0, Muir Glacier: 0, Berlin: 105235063, London: 105235063, Arrow Glacier: 105235063, Gray Glacier: 105235063, Terminal Total Difficulty: 0, Merge Netsplit: 105235063, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Engine: unknown}" genesis=0x2b955e7dd8df0ba8a50ee3e5c79065be9a370b1e52398694001067d40f7d6731
[INFO] [09-22|09:39:45.816] Initialising Ethereum protocol           network=10
[INFO] [09-22|09:39:45.816] Disk storage enabled for ethash DAGs     dir=/data/ethash-dags count=2
[INFO] [09-22|09:39:45.819] Starting private RPC server              on=0.0.0.0:9090
[INFO] [09-22|09:39:45.819] new subscription to logs established 
[INFO] [09-22|09:39:45.819] rpc filters: subscribing to Erigon events 
[INFO] [09-22|09:39:45.819] New txs subscriber joined 
[INFO] [09-22|09:39:45.819] new subscription to newHeaders established 
[INFO] [09-22|09:39:45.820] Reading JWT secret                       path=/config/jwtsecret.hex
[INFO] [09-22|09:39:45.820] HTTP endpoint opened for Engine API      url=[::]:8551 ws=true ws.compression=true
[INFO] [09-22|09:39:45.820] HTTP endpoint opened                     url=[::]:8545 ws=true ws.compression=true grpc=false
[INFO] [09-22|09:39:45.824] Started P2P networking                   version=67 self=enode://e3db89a27b8617bf947f0ba9906967257db752a911811368de9052582732caf5c939dafd4c8ba9a6e3849e92d8695f508c0057350186c4b8bf25cf40f52e48e3@127.0.0.1:0 name=erigon/v0.01.10/linux-amd64/go1.20.5
[INFO] [09-22|09:39:45.826] Started P2P networking                   version=68 self=enode://e3db89a27b8617bf947f0ba9906967257db752a911811368de9052582732caf5c939dafd4c8ba9a6e3849e92d8695f508c0057350186c4b8bf25cf40f52e48e3@127.0.0.1:0 name=erigon/v0.01.10/linux-amd64/go1.20.5
[INFO] [09-22|09:39:45.830] [2/15 Headers] Waiting for Consensus Layer... 
[INFO] [09-22|09:39:45.830] [txpool] Started 
[INFO] [09-22|09:40:45.819] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=23.7MB sys=44.1MB
[INFO] [09-22|09:41:45.816] [p2p] GoodPeers 
[INFO] [09-22|09:41:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=29.8MB sys=48.4MB
[INFO] [09-22|09:42:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=22.4MB sys=52.4MB
[INFO] [09-22|09:43:45.817] [p2p] GoodPeers 
[INFO] [09-22|09:43:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=27.7MB sys=52.5MB
[INFO] [09-22|09:44:45.819] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=20.8MB sys=52.5MB
[INFO] [09-22|09:45:45.817] [p2p] GoodPeers 
[INFO] [09-22|09:45:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=26.1MB sys=52.6MB
[INFO] [09-22|09:46:45.819] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=20.1MB sys=52.6MB
[INFO] [09-22|09:47:45.817] [p2p] GoodPeers 
[INFO] [09-22|09:47:45.819] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=26.1MB sys=52.6MB
[INFO] [09-22|09:48:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=20.9MB sys=52.6MB
[INFO] [09-22|09:49:45.817] [p2p] GoodPeers 
[INFO] [09-22|09:49:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=26.1MB sys=52.6MB
[INFO] [09-22|09:50:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=20.1MB sys=52.6MB
[INFO] [09-22|09:51:45.817] [p2p] GoodPeers 
[INFO] [09-22|09:51:45.819] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=26.1MB sys=52.6MB
[INFO] [09-22|09:52:45.820] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=20.1MB sys=52.6MB

(The logs that start with [INFO - entrypoint] are printed in custom entrypoint)

This is the command I used:

exec erigon --datadir=${DATA_DIR} \
  --rollup.sequencerhttp=${SEQUENCER_HTTP_URL} \
  --rollup.disabletxpoolgossip=true \
  --nodiscover \
  --http.addr=0.0.0.0 \
  --http.port=8545 \
  --http.corsdomain="*" \
  --http.vhosts="*" \
  --ws \
  --private.api.addr=0.0.0.0:9090 \
  --metrics \
  --metrics.addr=0.0.0.0 \
  --metrics.port=6060 \
  --port=${P2P_PORT} \
  --authrpc.jwtsecret=/config/jwtsecret.hex \
  --authrpc.addr 0.0.0.0 \
  --authrpc.vhosts=* \
  --authrpc.port=8551 \
  --chain=optimism-mainnet \
  ${EXTRA_FLAGS}
dappnodedev commented 1 year ago

Any ideas what could be causing this? failed to determine L2BlockRef of height 105235063, could not get payload: not found

ImTei commented 1 year ago

@dappnodedev 105235063 is the bedrock genesis block number of OP mainnet. so block 105235063 must be in the datadir that you downloaded from our link. can you double-check that you set the right datadir?

dappnodedev commented 1 year ago

@ImTei it seems like for some reason the data is not being recognized, but the path is the right one:

curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' http://op-erigon.dappnode:8545
{"jsonrpc":"2.0","id":1,"result":"0x0"}
dappnodedev commented 1 year ago 
ls /data/
LOCK       database   logs       nodes      temp
chaindata  jwt.hex    nodekey    snapshots  txpool
dappnodedev commented 1 year ago

As you can see in the output of the erigon process:

[INFO] [09-22|09:39:45.810] Opening Database                         label=chaindata path=/data/chaindata
pcw109550 commented 1 year ago

@ImTei it seems like for some reason the data is not being recognized, but the path is the right one:

curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' http://op-erigon.dappnode:8545
{"jsonrpc":"2.0","id":1,"result":"0x0"}

It seems that you have misconfigured your database. Its because the RPC must give 0x645c278 or 105235064 as latest block number.

pcw109550 commented 1 year ago 
ls /data/
LOCK       database   logs       nodes      temp
chaindata  jwt.hex    nodekey    snapshots  txpool

Can you check the content of /data/database? If you unpack our database, it creates directory database, NOT data. So you must fix your --datadir flag to /data/database, not /data.

dappnodedev commented 1 year ago

@ImTei it seems like for some reason the data is not being recognized, but the path is the right one:

curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' http://op-erigon.dappnode:8545
{"jsonrpc":"2.0","id":1,"result":"0x0"}

It seems that you have misconfigured your database. Its because the RPC must give 0x645c278 or 105235064 as latest block number.

Yes, that is why I posted the output, because it was so weird to have 0x0 as a result

dappnodedev commented 1 year ago

@pcw109550 You were right, setting --datadir to that path made the trick

Thanks so much to everyone!