Devnet broken for all versions

[imnotanoob]

Hi All,

All versions of optimism has devnet broken.

This is the error:

Fatal: Failed to register the Ethereum service: incompatible state scheme, stored: path, provided: hash
Fatal: Failed to register the Ethereum service: incompatible state scheme, stored: path, provided: hash

Looks like l2 is not able to connect with l1, causing l2 to break:

3ba208be5fb3   ops-bedrock-l2      "/bin/sh /entrypoint…"   57 seconds ago       Exited (1) 55 seconds ago                                                                                    ops-bedrock-l2-1

Looking at changes:

I believe there might be a dynamic dependency or something where previously it was working with Path, and now it isn't.

https://github.com/ethereum-optimism/optimism/commit/dae091f75731e90b6d4646adc4fa129645e56af1#diff-46560dcdaebe1cfcb3efb2042a1c4fdf8a504363bd3d0ca6a3fc85baa6e50296R49

this commit by @protolambda likely fixed it.

Can we backport this, or release a new version with this fixed? Devnet is very important to run integration tests against to make sure the underlying rpcs/network did not evolve

[tynes]

If you build from source on an old release, its still broken?

What release would you like this backported to?

[protolambda]

If you have an existing path local devnet db (like the log implies), then you need --state.scheme=path. We made it explicitly --state.scheme=hash because the hash scheme is required when using archive mode (for fault-proof proof generation and op-proposer output-root generation).

[imnotanoob]

If you start up any version from scratch. V18 or v19 I tested and do make devnet-up you will see this issue.

The changes @protolambda made fixed it. If we can just cut develop into a new patch release that would be enough in this case.

[imnotanoob]

If you have an existing path local devnet db (like the log implies), then you need --state.scheme=path. We made it explicitly --state.scheme=hash because the hash scheme is required when using archive mode (for fault-proof proof generation and op-proposer output-root generation).

I ran this in a brand new server as well as inside a docker container using DIND, so there was no existing path local devnet db

[samlaf]

on develop branch, I'm still running into issues where the op-node crashes:

docker ps -a
CONTAINER ID   IMAGE                                                                  COMMAND                  CREATED          STATUS                     PORTS                                                                            NAMES
5ed5d921ea16   us-docker.pkg.dev/oplabs-tools-artifacts/images/op-challenger:devnet   "op-challenger"          5 seconds ago    Up 5 seconds                                                                                                ops-bedrock-op-challenger-1
be4f5caf445d   us-docker.pkg.dev/oplabs-tools-artifacts/images/op-proposer:devnet     "op-proposer"            6 seconds ago    Up 5 seconds               0.0.0.0:6062->6060/tcp, 0.0.0.0:7302->7300/tcp, 0.0.0.0:6546->8545/tcp           ops-bedrock-op-proposer-1
8039d322e390   us-docker.pkg.dev/oplabs-tools-artifacts/images/op-batcher:devnet      "op-batcher"             6 seconds ago    Up 5 seconds               0.0.0.0:6061->6060/tcp, 0.0.0.0:7301->7300/tcp, 0.0.0.0:6545->8545/tcp           ops-bedrock-op-batcher-1
df34ba5fe086   nginx:1.25-alpine                                                      "/docker-entrypoint.…"   6 seconds ago    Up 5 seconds               0.0.0.0:8080->80/tcp                                                             ops-bedrock-artifact-server-1
fd7d268ab859   us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:devnet         "op-node --l1=ws://l…"   6 seconds ago    Exited (1) 5 seconds ago                                                                                    ops-bedrock-op-node-1
0d166d7a8cda   ops-bedrock-l2                                                         "/bin/sh /entrypoint…"   11 seconds ago   Up 10 seconds              8546/tcp, 30303/tcp, 30303/udp, 0.0.0.0:8060->6060/tcp, 0.0.0.0:9545->8545/tcp   ops-bedrock-l2-1
9ba3b88f72ea   ops-bedrock-l1-vc                                                      "/bin/sh /entrypoint…"   12 seconds ago   Up 12 seconds                                                                                               ops-bedrock-l1-vc-1
e3267dcf7929   ops-bedrock-l1-bn                                                      "/bin/sh /entrypoint…"   12 seconds ago   Up 12 seconds              0.0.0.0:5052->5052/tcp, 0.0.0.0:9000->9000/tcp                                   ops-bedrock-l1-bn-1
f918bc29a982   ops-bedrock-l1                                                         "/bin/bash /entrypoi…"   12 seconds ago   Up 12 seconds              0.0.0.0:8545-8546->8545-8546/tcp, 30303/tcp, 30303/udp, 0.0.0.0:7060->6060/tcp   ops-bedrock-l1-1

with op-node logs being:

docker logs fd7d268ab859
t=2024-08-10T02:05:18+0000 lvl=warn msg="Found a deprecated flag which will be removed in a future version" flag_name=p2p.scoring.peers
t=2024-08-10T02:05:18+0000 lvl=info msg="Not opted in to ProtocolVersions signal loading, disabling ProtocolVersions contract now."
t=2024-08-10T02:05:19+0000 lvl=info msg="No persisted sequencer state loaded"
t=2024-08-10T02:05:19+0000 lvl=info msg="Rollup Config" l2_chain_id=901 l2_network="unknown L2" l1_chain_id=900 l1_network="unknown L1" l2_start_time=1723177525 l2_block_hash=0x2d6440c06e0837bc8c9d3a1b899839918611e7dace7bc5ec26c33e7eaf3fbf5e l2_block_number=0 l1_block_hash=0xd1c1fa8f05dba47ceffc8ad00eceb17ba102516214301b3c83ada6438ca69791 l1_block_number=1 regolith_time="@ genesis" canyon_time="@ genesis" delta_time="@ genesis" ecotone_time="@ genesis" fjord_time="@ genesis" granite_time="(not configured)" holocene_time="(not configured)" interop_time="(not configured)" plasma_mode=false
t=2024-08-10T02:05:19+0000 lvl=info msg="Initializing rollup node" version=v0.0.0-773e476b-1723243615
t=2024-08-10T02:05:19+0000 lvl=error msg="Error initializing the rollup node" err="failed to init L1: failed to validate the L1 config: failed to get L1 genesis blockhash: failed to fetch header by num 1: not found"
t=2024-08-10T02:05:19+0000 lvl=crit msg="Application failed" message="failed to setup: unable to create the rollup node: failed to init L1: failed to validate the L1 config: failed to get L1 genesis blockhash: failed to fetch header by num 1: not found"

Any idea what's going on?

EDIT: nvm, seems fixed after make devnet-clean. There was probably some old config that wasn't properly getting overwritten?

[imnotanoob]

@tynes , @protolambda - any chance we can cut this into a new release branch?

[tynes]

@imnotanoob We will be doing a release sometime soon. For now you should be able to use a specific commit on develop that includes the fix

[imnotanoob]

@imnotanoob We will be doing a release sometime soon. For now you should be able to use a specific commit on develop that includes the fix

Thanks, yep doing that for now.

[avishkarabhishek786]

I get this error when I tried with develop and v1.9.0. In develop there is no package.json either.

git submodule update --init --recursive make[1]: Entering directory '/home/aks/Documents/optimism/optimism' ./ops/scripts/geth-version-checker.sh && \ (echo "Geth versions match, not installing geth..."; true) || \ (echo "Versions do not match, installing geth!"; \ go install -v github.com/ethereum/go-ethereum/cmd/geth@v1.13.14; \ echo "Installed geth!"; true) ./ops/scripts/geth-version-checker.sh: line 7: geth: command not found Geth version does not match! Local geth version: v Expected geth version: v1.13.14-stable Versions do not match, installing geth! // github.com/ethereum/go-ethereum/cmd/geth link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld Installed geth! make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' go install -v github.com/protolambda/eth2-testnet-genesis@v0.10.0 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' make -C ./op-program op-program make[2]: Entering directory '/home/aks/Documents/optimism/optimism/op-program' env GO111MODULE=on GOOS= GOARCH= CGO_ENABLED=0 go build -v -ldflags "-X main.GitCommit=ec45f6634ab2855a4ae5d30c4e240d79f081d689 -X main.GitDate=1723023640 -X github.com/ethereum-optimism/optimism/op-program/version.Version=v0.0.0 -X github.com/ethereum-optimism/optimism/op-program/version.Meta=" -o ./bin/op-program ./host/cmd/main.go //command-line-arguments link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld make[2]: [Makefile:24: op-program-host] Error 1 make[2]: Leaving directory '/home/aks/Documents/optimism/optimism/op-program' make[1]: [Makefile:129: op-program] Error 2 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make: *** [Makefile:166: pre-devnet] Error 2

[avishkarabhishek786]

I get this error when I tried with develop and v1.9.0. In develop there is no package.json either.

git submodule update --init --recursive make[1]: Entering directory '/home/aks/Documents/optimism/optimism' ./ops/scripts/geth-version-checker.sh && (echo "Geth versions match, not installing geth..."; true) || (echo "Versions do not match, installing geth!"; go install -v github.com/ethereum/go-ethereum/cmd/geth@v1.13.14; echo "Installed geth!"; true) ./ops/scripts/geth-version-checker.sh: line 7: geth: command not found Geth version does not match! Local geth version: v Expected geth version: v1.13.14-stable Versions do not match, installing geth! // github.com/ethereum/go-ethereum/cmd/geth link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld Installed geth! make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' go install -v github.com/protolambda/eth2-testnet-genesis@v0.10.0 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' make -C ./op-program op-program make[2]: Entering directory '/home/aks/Documents/optimism/optimism/op-program' env GO111MODULE=on GOOS= GOARCH= CGO_ENABLED=0 go build -v -ldflags "-X main.GitCommit=ec45f6634ab2855a4ae5d30c4e240d79f081d689 -X main.GitDate=1723023640 -X github.com/ethereum-optimism/optimism/op-program/version.Version=v0.0.0 -X github.com/ethereum-optimism/optimism/op-program/version.Meta=" -o ./bin/op-program ./host/cmd/main.go //command-line-arguments link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld make[2]: [Makefile:24: op-program-host] Error 1 make[2]: Leaving directory '/home/aks/Documents/optimism/optimism/op-program' make[1]: [Makefile:129: op-program] Error 2 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make: *** [Makefile:166: pre-devnet] Error 2

UPDATE

So my Go version was 1.23.0. When I switched to 1.21.13 things started working for me. I guess the reason is mentioned in memsize github - "NOTE: As of Go 1.23, memsize no longer works because of a restriction added by the Go toolchain."

However, I am now getting a new error -

eth2-testnet-genesis path: l1-generate-beacon-genesis.sh: 7: eth2-testnet-genesis: not found Traceback (most recent call last): File "/home/aks/Documents/optimism/optimism/./bedrock-devnet/main.py", line 9, in <module> main() File "/home/aks/Documents/optimism/optimism/./bedrock-devnet/main.py", line 5, in main devnet.main() File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 118, in main devnet_deploy(paths) File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 201, in devnet_deploy run_command([ File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 365, in run_command return subprocess.run( File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['sh', 'l1-generate-beacon-genesis.sh']' returned non-zero exit status 127. make: *** [Makefile:177: devnet-up] Error 1

Full log

[samlaf]

I get this error when I tried with develop and v1.9.0. In develop there is no package.json either. git submodule update --init --recursive make[1]: Entering directory '/home/aks/Documents/optimism/optimism' ./ops/scripts/geth-version-checker.sh && (echo "Geth versions match, not installing geth..."; true) || (echo "Versions do not match, installing geth!"; go install -v github.com/ethereum/go-ethereum/cmd/geth@v1.13.14; echo "Installed geth!"; true) ./ops/scripts/geth-version-checker.sh: line 7: geth: command not found Geth version does not match! Local geth version: v Expected geth version: v1.13.14-stable Versions do not match, installing geth! // github.com/ethereum/go-ethereum/cmd/geth link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld Installed geth! make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' go install -v github.com/protolambda/eth2-testnet-genesis@v0.10.0 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' make -C ./op-program op-program make[2]: Entering directory '/home/aks/Documents/optimism/optimism/op-program' env GO111MODULE=on GOOS= GOARCH= CGO_ENABLED=0 go build -v -ldflags "-X main.GitCommit=ec45f6634ab2855a4ae5d30c4e240d79f081d689 -X main.GitDate=1723023640 -X github.com/ethereum-optimism/optimism/op-program/version.Version=v0.0.0 -X github.com/ethereum-optimism/optimism/op-program/version.Meta=" -o ./bin/op-program ./host/cmd/main.go //command-line-arguments link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld make[2]: [Makefile:24: op-program-host] Error 1 make[2]: Leaving directory '/home/aks/Documents/optimism/optimism/op-program' make[1]: [Makefile:129: op-program] Error 2 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make: *** [Makefile:166: pre-devnet] Error 2

UPDATE

So my Go version was 1.23.0. When I switched to 1.21.13 things started working for me. I guess the reason is mentioned in memsize github - "NOTE: As of Go 1.23, memsize no longer works because of a restriction added by the Go toolchain."

However, I am now getting a new error -

eth2-testnet-genesis path: l1-generate-beacon-genesis.sh: 7: eth2-testnet-genesis: not found Traceback (most recent call last): File "/home/aks/Documents/optimism/optimism/./bedrock-devnet/main.py", line 9, in <module> main() File "/home/aks/Documents/optimism/optimism/./bedrock-devnet/main.py", line 5, in main devnet.main() File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 118, in main devnet_deploy(paths) File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 201, in devnet_deploy run_command([ File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 365, in run_command return subprocess.run( File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['sh', 'l1-generate-beacon-genesis.sh']' returned non-zero exit status 127. make: *** [Makefile:177: devnet-up] Error 1

Full log

Think you just need to install eth2-testnet-genesis tool. look it up it has its own GitHub repo you can install it via go toolchain.

[imnotanoob]

@imnotanoob We will be doing a release sometime soon. For now you should be able to use a specific commit on develop that includes the fix

Hey tynes. Just wanted to follow up :)

[michalsidzej]

I get this error when I tried with develop and v1.9.0. In develop there is no package.json either. git submodule update --init --recursive make[1]: Entering directory '/home/aks/Documents/optimism/optimism' ./ops/scripts/geth-version-checker.sh && (echo "Geth versions match, not installing geth..."; true) || (echo "Versions do not match, installing geth!"; go install -v github.com/ethereum/go-ethereum/cmd/geth@v1.13.14; echo "Installed geth!"; true) ./ops/scripts/geth-version-checker.sh: line 7: geth: command not found Geth version does not match! Local geth version: v Expected geth version: v1.13.14-stable Versions do not match, installing geth! // github.com/ethereum/go-ethereum/cmd/geth link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld Installed geth! make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' go install -v github.com/protolambda/eth2-testnet-genesis@v0.10.0 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' make -C ./op-program op-program make[2]: Entering directory '/home/aks/Documents/optimism/optimism/op-program' env GO111MODULE=on GOOS= GOARCH= CGO_ENABLED=0 go build -v -ldflags "-X main.GitCommit=ec45f6634ab2855a4ae5d30c4e240d79f081d689 -X main.GitDate=1723023640 -X github.com/ethereum-optimism/optimism/op-program/version.Version=v0.0.0 -X github.com/ethereum-optimism/optimism/op-program/version.Meta=" -o ./bin/op-program ./host/cmd/main.go //command-line-arguments link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld make[2]: [Makefile:24: op-program-host] Error 1 make[2]: Leaving directory '/home/aks/Documents/optimism/optimism/op-program' make[1]: [Makefile:129: op-program] Error 2 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make: *** [Makefile:166: pre-devnet] Error 2

UPDATE So my Go version was 1.23.0. When I switched to 1.21.13 things started working for me. I guess the reason is mentioned in memsize github - "NOTE: As of Go 1.23, memsize no longer works because of a restriction added by the Go toolchain." However, I am now getting a new error - eth2-testnet-genesis path: l1-generate-beacon-genesis.sh: 7: eth2-testnet-genesis: not found Traceback (most recent call last): File "/home/aks/Documents/optimism/optimism/./bedrock-devnet/main.py", line 9, in <module> main() File "/home/aks/Documents/optimism/optimism/./bedrock-devnet/main.py", line 5, in main devnet.main() File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 118, in main devnet_deploy(paths) File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 201, in devnet_deploy run_command([ File "/home/aks/Documents/optimism/optimism/bedrock-devnet/devnet/__init__.py", line 365, in run_command return subprocess.run( File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['sh', 'l1-generate-beacon-genesis.sh']' returned non-zero exit status 127. make: *** [Makefile:177: devnet-up] Error 1 Full log

Think you just need to install eth2-testnet-genesis tool. look it up it has its own GitHub repo you can install it via go toolchain.

Thank you for that. I was struggling for a long time to find the problem, as make devnet-up would actually try starting the devnet without eth2-testnet-genesis (I did not set PATH="$PATH:$HOME/go/bin").

First run would error with eth2-testnet-genesis: command not found. The second run would start the devnet, but error with "fetching start block by number: operation failed permanently after 24 attempts: not found".

This was happening because the L1 was still at block 0, as no contracts were deployed. You can confirm that by running

curl -X POST -H "Content-Type: application/json" --data  \
  '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
 localhost:8545

The problem was lack of eth2-genesis-testnet command, but I'm not sure why it does run on the second try.

The fix for me was adding export PATH="$PATH:$HOME/go/bin" to ~/.zshrc. Hope it helps someone.

[nic225]

Hi, I was having a similar issue where the "First run would error with eth2-testnet-genesis: command not found. The second run would start the devnet, but error with "fetching start block by number: operation failed permanently after 24 attempts: not found"."

I used your solution of adding export PATH="$PATH:$HOME/go/bin" to the .bashrc but now I am getting a new error. Upon the first make devnet-up run.

For note I am running on ubuntu ubuntu 24.04.1 with go version go1.23.2 linux/amd64.

Have you seen this error:
l2_output_oracle = addresses['L2OutputOracleProxy']

KeyError: 'L2OutputOracleProxy'

@samlaf @tynes @imnotanoob @michalsidzej 

Thanks for the help.

[samlaf]

@nic225 that's a new regression: see https://github.com/ethereum-optimism/optimism/issues/12354

[Tron2024-cpu]

I get this error when I tried with develop and v1.9.0. In develop there is no package.json either.

git submodule update --init --recursive make[1]: Entering directory '/home/aks/Documents/optimism/optimism' ./ops/scripts/geth-version-checker.sh && (echo "Geth versions match, not installing geth..."; true) || (echo "Versions do not match, installing geth!"; go install -v github.com/ethereum/go-ethereum/cmd/geth@v1.13.14; echo "Installed geth!"; true) ./ops/scripts/geth-version-checker.sh: line 7: geth: command not found Geth version does not match! Local geth version: v Expected geth version: v1.13.14-stable Versions do not match, installing geth! // github.com/ethereum/go-ethereum/cmd/geth link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld Installed geth! make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' go install -v github.com/protolambda/eth2-testnet-genesis@v0.10.0 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make[1]: Entering directory '/home/aks/Documents/optimism/optimism' make -C ./op-program op-program make[2]: Entering directory '/home/aks/Documents/optimism/optimism/op-program' env GO111MODULE=on GOOS= GOARCH= CGO_ENABLED=0 go build -v -ldflags "-X main.GitCommit=ec45f6634ab2855a4ae5d30c4e240d79f081d689 -X main.GitDate=1723023640 -X github.com/ethereum-optimism/optimism/op-program/version.Version=v0.0.0 -X github.com/ethereum-optimism/optimism/op-program/version.Meta=" -o ./bin/op-program ./host/cmd/main.go //command-line-arguments link: github.com/fjl/memsize: invalid reference to runtime.stopTheWorld make[2]: [Makefile:24: op-program-host] Error 1 make[2]: Leaving directory '/home/aks/Documents/optimism/optimism/op-program' make[1]: [Makefile:129: op-program] Error 2 make[1]: Leaving directory '/home/aks/Documents/optimism/optimism' make: *** [Makefile:166: pre-devnet] Error 2

Never versión have a package json. How You can resolver this ???