search

ethpandaops / ethereum-package

A Kurtosis package that deploys a private, portable, and modular Ethereum devnet
MIT License
194 stars 99 forks source link

dial tcp 172.16.0.13:9000: i/o timeout #694

Open davidLeeeeeeeeeeee opened 1 week ago

davidLeeeeeeeeeeee commented 1 week ago

Ubuntu 22.04 LTS

Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3

kurtosis-cli version (0.90.1)

(base) root@ST-ZY-28:/var/zk_evm# kurtosis run github.com/ethpandaops/ethereum-package --args-file ./network_params2.yaml --image-download always
INFO[2024-06-27T07:36:57+08:00] Creating a new enclave for Starlark to run inside... 
INFO[2024-06-27T07:37:03+08:00] Enclave 'dry-geyser' created successfully    

Container images used in this run:
> ethereum/client-go:latest - remotely downloaded
> ethpandaops/ethereum-genesis-generator:3.3.5 - remotely downloaded
> python:3.11-alpine - remotely downloaded
> protolambda/eth2-val-tools:latest - remotely downloaded
> badouralix/curl-jq - remotely downloaded
> consensys/teku:latest - remotely downloaded
> ethpandaops/dora:latest - remotely downloaded
> ethpandaops/lighthouse:stable - remotely downloaded

Uploading file '/static_files/jwt/jwtsecret' to files artifact 'jwt_file'
Files with artifact name 'jwt_file' uploaded with artifact UUID 'bda9564aa9874d3eb00d151b57365786'

Uploading file '/static_files/keymanager/keymanager.txt' to files artifact 'keymanager_file'
Files with artifact name 'keymanager_file' uploaded with artifact UUID 'e7799dd9ca6847f9b1a4e007738d1209'

Printing a message
Read the prometheus, grafana templates

Printing a message
Launching participant network with 3 participants and the following network params struct(additional_preloaded_contracts = {}, churn_limit_quotient = 65536, custody_requirement = 1, data_column_sidecar_subnet_count = 32, deneb_fork_epoch = 0, deposit_contract_address = "0x4242424242424242424242424242424242424242", devnet_repo = "ethpandaops", eip7594_fork_epoch = 100000001, eip7594_fork_version = "0x70000038", ejection_balance = 16000000000, electra_fork_epoch = 100000000, eof_activation_epoch = "", eth1_follow_distance = 2048, genesis_delay = 20, max_per_epoch_activation_churn_limit = 8, min_validator_withdrawability_delay = 256, network = "kurtosis", network_id = "585858", network_sync_base_url = "https://ethpandaops-ethereum-node-snapshots.ams3.cdn.digitaloceanspaces.com/", num_validator_keys_per_node = 64, preregistered_validator_count = 0, preregistered_validator_keys_mnemonic = "giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete", preset = "mainnet", samples_per_slot = 8, seconds_per_slot = 12, shard_committee_period = 256, target_number_of_peers = 70)

Printing a message
Generating cl validator key stores

Adding service with name 'validator-key-generation-cl-validator-keystore' and image 'protolambda/eth2-val-tools:latest'
Service 'validator-key-generation-cl-validator-keystore' added with service UUID '154ff54a408d40af813a95b837989070'

Generating keystores
Command returned with exit code '0' with no output

Verifying whether two values meet a certain condition '=='
Verification succeeded. Value is '0'.

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/node-0-keystores/' to files artifact with name '1-lighthouse-geth-0-63-0'
Files with artifact name '1-lighthouse-geth-0-63-0' uploaded with artifact UUID 'bb406afedb3a433da318e2d887bc74cd'

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/node-1-keystores/' to files artifact with name '2-lighthouse-geth-64-127-0'
Files with artifact name '2-lighthouse-geth-64-127-0' uploaded with artifact UUID '0c1bef28eb784981972f6a411fdca353'

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/node-2-keystores/' to files artifact with name '3-teku-geth-128-191-0'
Files with artifact name '3-teku-geth-128-191-0' uploaded with artifact UUID 'a86050e797794aeea32fe31d6e415ab7'

Storing prysm password in a file
Command returned with exit code '0' with no output

Verifying whether two values meet a certain condition '=='
Verification succeeded. Value is '0'.

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/tmp/prysm-password.txt' to files artifact with name 'prysm-password'
Files with artifact name 'prysm-password' uploaded with artifact UUID '36c3d12fe1aa467da10e41f4c1ca1a99'

Printing a message
{
        "per_node_keystores": [
                {
                        "files_artifact_uuid": "1-lighthouse-geth-0-63-0",
                        "nimbus_keys_relative_dirpath": "/nimbus-keys",
                        "prysm_relative_dirpath": "/prysm",
                        "raw_keys_relative_dirpath": "/keys",
                        "raw_root_dirpath": "",
                        "raw_secrets_relative_dirpath": "/secrets",
                        "teku_keys_relative_dirpath": "/teku-keys",
                        "teku_secrets_relative_dirpath": "/teku-secrets"
                },
                {
                        "files_artifact_uuid": "2-lighthouse-geth-64-127-0",
                        "nimbus_keys_relative_dirpath": "/nimbus-keys",
                        "prysm_relative_dirpath": "/prysm",
                        "raw_keys_relative_dirpath": "/keys",
                        "raw_root_dirpath": "",
                        "raw_secrets_relative_dirpath": "/secrets",
                        "teku_keys_relative_dirpath": "/teku-keys",
                        "teku_secrets_relative_dirpath": "/teku-secrets"
                },
                {
                        "files_artifact_uuid": "3-teku-geth-128-191-0",
                        "nimbus_keys_relative_dirpath": "/nimbus-keys",
                        "prysm_relative_dirpath": "/prysm",
                        "raw_keys_relative_dirpath": "/keys",
                        "raw_root_dirpath": "",
                        "raw_secrets_relative_dirpath": "/secrets",
                        "teku_keys_relative_dirpath": "/teku-keys",
                        "teku_secrets_relative_dirpath": "/teku-secrets"
                }
        ],
        "prysm_password_artifact_uuid": "prysm-password",
        "prysm_password_relative_filepath": "prysm-password.txt"
}

Getting final genesis timestamp
Command returned with exit code '0' and the following output: 1719445084

Printing a message
Generating EL CL data

Rendering a template to a files artifact with name 'genesis-el-cl-env-file'
Templates artifact name 'genesis-el-cl-env-file' rendered with artifact UUID '6e53c4476eb241de9e230965012265f1'

Creating genesis
Command returned with exit code '0' and the following output:
--------------------
+ '[' -f /data/metadata/genesis.json ']'
++ mktemp -d -t ci-XXXXXXXXXX
+ tmp_dir=/tmp/ci-JVD5dzzsSr
+ mkdir -p /data/metadata
+ envsubst
+ python3 /apps/el-gen/genesis_geth.py /tmp/ci-JVD5dzzsSr/genesis-config.yaml
+ python3 /apps/el-gen/genesis_chainspec.py /tmp/ci-JVD5dzzsSr/genesis-config.yaml
+ python3 /apps/el-gen/genesis_besu.py /tmp/ci-JVD5dzzsSr/genesis-config.yaml
+ gen_cl_config
+ . /apps/el-gen/.venv/bin/activate
++ deactivate nondestructive
++ '[' -n /root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ']'
++ PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ unset _OLD_VIRTUAL_PATH
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ unset VIRTUAL_ENV_PROMPT
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/apps/el-gen/.venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/apps/el-gen/.venv/bin:/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1='(.venv) '
++ PS1='(.venv) (.venv) '
++ export PS1
++ VIRTUAL_ENV_PROMPT='(.venv) '
++ export VIRTUAL_ENV_PROMPT
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ set -x
+ '[' -f /data/metadata/genesis.ssz ']'
++ mktemp -d -t ci-XXXXXXXXXX
+ tmp_dir=/tmp/ci-D9DSveuswY
+ mkdir -p /data/metadata
+ mkdir -p /data/parsed
+ envsubst
+ envsubst
+ [[ mainnet == \m\i\n\i\m\a\l ]]
+ cp /tmp/ci-D9DSveuswY/mnemonics.yaml /data/metadata/mnemonics.yaml
+ grep DEPOSIT_CONTRACT_ADDRESS /data/metadata/config.yaml
+ cut -d ' ' -f2
+ echo 0
+ echo enr:-Iq4QJk4WqRkjsX5c2CXtOra6HnxN-BMXnWhmhEQO9Bn9iABTJGdjUOurM7Btj1ouKaFkvTRoju5vz2GPmVON2dffQKGAX53x8JigmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCCIyk
+ envsubst
+ genesis_args=(deneb --config /data/metadata/config.yaml --mnemonics $tmp_dir/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 $PRESET_BASE --preset-altair $PRESET_BASE --preset-bellatrix $PRESET_BASE --preset-capella $PRESET_BASE --preset-deneb $PRESET_BASE)
+ [[ 0x00 == \0\x\0\1 ]]
+ [[ '' != '' ]]
+ [[ '' != '' ]]
+ genesis_args+=(--eth1-config /data/metadata/genesis.json)
+ '[' -z '' ']'
+ zcli_args=(pretty deneb BeaconState --preset-phase0 $PRESET_BASE --preset-altair $PRESET_BASE --preset-bellatrix $PRESET_BASE --preset-capella $PRESET_BASE --preset-deneb $PRESET_BASE /data/metadata/genesis.ssz)
+ /usr/local/bin/eth2-testnet-genesis deneb --config /data/metadata/config.yaml --mnemonics /tmp/ci-D9DSveuswY/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet --eth1-config /data/metadata/genesis.json
zrnt version: v0.32.3
Using CL MIN_GENESIS_TIME for genesis timestamp
processing mnemonic 0, for 192 validators
...validator 100/192
Writing pubkeys list file...
generated 192 validators from mnemonic yaml (/tmp/ci-D9DSveuswY/mnemonics.yaml)
eth2 genesis at 1719445084 + 20 = 1719445104  (2024-06-26 23:38:24 +0000 UTC)
done preparing state, serializing SSZ now...
done!
+ /usr/local/bin/zcli pretty deneb BeaconState --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet /data/metadata/genesis.ssz
+ echo 'Genesis args: deneb' --config /data/metadata/config.yaml --mnemonics /tmp/ci-D9DSveuswY/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet --eth1-config /data/metadata/genesis.json
Genesis args: deneb --config /data/metadata/config.yaml --mnemonics /tmp/ci-D9DSveuswY/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet --eth1-config /data/metadata/genesis.json
++ jq -r .latest_execution_payload_header.block_number /data/parsed/parsedConsensusGenesis.json
Genesis block number: 0
+ echo 'Genesis block number: 0'
++ jq -r .latest_execution_payload_header.block_hash /data/parsed/parsedConsensusGenesis.json
Genesis block hash: 0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1
+ echo 'Genesis block hash: 0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1'
+ jq -r .eth1_data.block_hash /data/parsed/parsedConsensusGenesis.json
+ tr -d '\n'
+ jq -r .genesis_validators_root /data/parsed/parsedConsensusGenesis.json
+ tr -d '\n'
+ gen_shared_files
+ . /apps/el-gen/.venv/bin/activate
++ deactivate nondestructive
++ '[' -n /root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ']'
++ PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ unset _OLD_VIRTUAL_PATH
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '(.venv) ' ']'
++ PS1='(.venv) '
++ export PS1
++ unset _OLD_VIRTUAL_PS1
++ unset VIRTUAL_ENV
++ unset VIRTUAL_ENV_PROMPT
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/apps/el-gen/.venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/apps/el-gen/.venv/bin:/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1='(.venv) '
++ PS1='(.venv) (.venv) '
++ export PS1
++ VIRTUAL_ENV_PROMPT='(.venv) '
++ export VIRTUAL_ENV_PROMPT
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ set -x
+ mkdir -p /data/metadata
+ '[' -f /data/jwt/jwtsecret ']'
+ mkdir -p /data/jwt
++ openssl rand -hex 32
++ tr -d '\n'
+ echo -n 0xaab62aed5f4914d74de359ce8c327f2ead6685ffbbc062ea793f0a3e7ce93cff
+ '[' -f /data/metadata/genesis.json ']'
++ cat /data/metadata/genesis.json
++ jq -r '.config.terminalTotalDifficulty | tostring'
+ terminalTotalDifficulty=0
+ sed -i 's/TERMINAL_TOTAL_DIFFICULTY:.*/TERMINAL_TOTAL_DIFFICULTY: 0/' /data/metadata/config.yaml
+ '[' false = true ']'

--------------------

Reading genesis validators root
Command returned with exit code '0' and the following output: 0xa63a1b2ffefb0fea452e9ad530c9a4bbd58c97c6ad82da001f142be64c91d61b

Reading prague time from genesis
Command returned with exit code '0' and the following output: 40119445104

Adding service with name 'el-1-geth-lighthouse' and image 'ethereum/client-go:latest'
Service 'el-1-geth-lighthouse' added with service UUID '294aacfef50642ff9b63bd3f9e45309c'

Waiting for at most '15m0s' for service 'el-1-geth-lighthouse' to reach a certain state
Wait took 1 tries (38.156124ms in total). Assertion passed with following:
Request had response code '200' and body "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"id\":\"0015779b9c938aa4c39453251c0987901cdebd498506297aeb696fc34c2a7459\",\"name\":\"Geth/v1.14.6-unstable-9298d2db/linux-amd64/go1.22.4\",\"enode\":\"enode://68f23d016ec17b6cdaaff13c9b867051289ea51846a8dd132a74c9d4720a18adcfbc33ebbb9647a25812104aeee6013ba014f0581ecd41c5b913281bf628eb68@172.16.0.10:30303\",\"enr\":\"enr:-Ki4QCZ_HN4PY05VtnQhg2dRk_7JA7TIvHAOej_vh0J3naakPUMvHJhxPakQtr9QI9n64INgX1OdOvuJQ3xHO9ttYBiGAZBW6dmHg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAColzZWNwMjU2azGhAmjyPQFuwXts2q_xPJuGcFEonqUYRqjdEyp0ydRyChithHNuYXDAg3RjcIJ2X4N1ZHCCdl8\",\"ip\":\"172.16.0.10\",\"ports\":{\"discovery\":30303,\"listener\":30303},\"listenAddr\":\"[::]:30303\",\"protocols\":{\"eth\":{\"network\":585858,\"difficulty\":1,\"genesis\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\",\"config\":{\"chainId\":585858,\"homesteadBlock\":0,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"mergeNetsplitBlock\":0,\"shanghaiTime\":0,\"cancunTime\":0,\"pragueTime\":40119445104,\"terminalTotalDifficulty\":0,\"terminalTotalDifficultyPassed\":true},\"head\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\"},\"snap\":{}}}}\n", with extracted fields:
'extract.enr': "enr:-Ki4QCZ_HN4PY05VtnQhg2dRk_7JA7TIvHAOej_vh0J3naakPUMvHJhxPakQtr9QI9n64INgX1OdOvuJQ3xHO9ttYBiGAZBW6dmHg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAColzZWNwMjU2azGhAmjyPQFuwXts2q_xPJuGcFEonqUYRqjdEyp0ydRyChithHNuYXDAg3RjcIJ2X4N1ZHCCdl8"
'extract.enode': "enode://68f23d016ec17b6cdaaff13c9b867051289ea51846a8dd132a74c9d4720a18adcfbc33ebbb9647a25812104aeee6013ba014f0581ecd41c5b913281bf628eb68@172.16.0.10:30303"

Adding service with name 'el-2-geth-lighthouse' and image 'ethereum/client-go:latest'
Service 'el-2-geth-lighthouse' added with service UUID '21e3516db10e4e69ae126cae31086e24'

Waiting for at most '15m0s' for service 'el-2-geth-lighthouse' to reach a certain state
Wait took 1 tries (54.889823ms in total). Assertion passed with following:
Request had response code '200' and body "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"id\":\"c0cee76f0fea4deb74e3def28ec53eab34447f09bc18914a270b0b589daf2b02\",\"name\":\"Geth/v1.14.6-unstable-9298d2db/linux-amd64/go1.22.4\",\"enode\":\"enode://4b59e59dcac69c2ab50155e9a14e6caaf72f8da38685f666ec12f8ba21c2a737cca41295a15034c05be1acd1883b1dc255050563d4b1a46d676d3c6427a752b0@172.16.0.11:30303\",\"enr\":\"enr:-Ki4QNzXjqxeBjdql6Lpwo2PJbY4ASRBH3R_mPwJ57_pDt4rGLFaBeQB5BNp9LKUtxc0g9bYJnOYkb5UnD3TvxczYDiGAZBW6fD8g2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAC4lzZWNwMjU2azGhAktZ5Z3KxpwqtQFV6aFObKr3L42jhoX2ZuwS-Lohwqc3hHNuYXDAg3RjcIJ2X4N1ZHCCdl8\",\"ip\":\"172.16.0.11\",\"ports\":{\"discovery\":30303,\"listener\":30303},\"listenAddr\":\"[::]:30303\",\"protocols\":{\"eth\":{\"network\":585858,\"difficulty\":1,\"genesis\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\",\"config\":{\"chainId\":585858,\"homesteadBlock\":0,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"mergeNetsplitBlock\":0,\"shanghaiTime\":0,\"cancunTime\":0,\"pragueTime\":40119445104,\"terminalTotalDifficulty\":0,\"terminalTotalDifficultyPassed\":true},\"head\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\"},\"snap\":{}}}}\n", with extracted fields:
'extract.enr': "enr:-Ki4QNzXjqxeBjdql6Lpwo2PJbY4ASRBH3R_mPwJ57_pDt4rGLFaBeQB5BNp9LKUtxc0g9bYJnOYkb5UnD3TvxczYDiGAZBW6fD8g2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAC4lzZWNwMjU2azGhAktZ5Z3KxpwqtQFV6aFObKr3L42jhoX2ZuwS-Lohwqc3hHNuYXDAg3RjcIJ2X4N1ZHCCdl8"
'extract.enode': "enode://4b59e59dcac69c2ab50155e9a14e6caaf72f8da38685f666ec12f8ba21c2a737cca41295a15034c05be1acd1883b1dc255050563d4b1a46d676d3c6427a752b0@172.16.0.11:30303"

Adding service with name 'el-3-geth-teku' and image 'ethereum/client-go:latest'
Service 'el-3-geth-teku' added with service UUID '1392da84cc1e49889f462480cf5a3e4b'

Waiting for at most '15m0s' for service 'el-3-geth-teku' to reach a certain state
Wait took 1 tries (46.107772ms in total). Assertion passed with following:
Request had response code '200' and body "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"id\":\"b5b793c6865d58156f3d1019c50897f5a9dd12946f4a69cb4eb928ed0b69c73b\",\"name\":\"Geth/v1.14.6-unstable-9298d2db/linux-amd64/go1.22.4\",\"enode\":\"enode://bf7dff3e8fc30d7496cb084871203135063ffe17c3b5b467755428577dadf7a0b17cbc157e9e1edbf0dadcdc6c4e2a6a73cec2e2a34774379d7fc77b8b089462@172.16.0.12:30303\",\"enr\":\"enr:-Ki4QOgwbxm8dvivQ0HhHq3sVqt5BFwGw8R4lNMSrvoGzlKhLzPe82D_UegFkJEMIeH8GZmdSqzrrJxlT73NxdXjnSOGAZBW6gjgg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAADIlzZWNwMjU2azGhAr99_z6Pww10lssISHEgMTUGP_4Xw7W0Z3VUKFd9rfeghHNuYXDAg3RjcIJ2X4N1ZHCCdl8\",\"ip\":\"172.16.0.12\",\"ports\":{\"discovery\":30303,\"listener\":30303},\"listenAddr\":\"[::]:30303\",\"protocols\":{\"eth\":{\"network\":585858,\"difficulty\":1,\"genesis\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\",\"config\":{\"chainId\":585858,\"homesteadBlock\":0,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"mergeNetsplitBlock\":0,\"shanghaiTime\":0,\"cancunTime\":0,\"pragueTime\":40119445104,\"terminalTotalDifficulty\":0,\"terminalTotalDifficultyPassed\":true},\"head\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\"},\"snap\":{}}}}\n", with extracted fields:
'extract.enode': "enode://bf7dff3e8fc30d7496cb084871203135063ffe17c3b5b467755428577dadf7a0b17cbc157e9e1edbf0dadcdc6c4e2a6a73cec2e2a34774379d7fc77b8b089462@172.16.0.12:30303"
'extract.enr': "enr:-Ki4QOgwbxm8dvivQ0HhHq3sVqt5BFwGw8R4lNMSrvoGzlKhLzPe82D_UegFkJEMIeH8GZmdSqzrrJxlT73NxdXjnSOGAZBW6gjgg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAADIlzZWNwMjU2azGhAr99_z6Pww10lssISHEgMTUGP_4Xw7W0Z3VUKFd9rfeghHNuYXDAg3RjcIJ2X4N1ZHCCdl8"

Printing a message
Successfully added 3 EL participants

Printing a message
Launching CL network

Adding service with name 'cl-1-lighthouse-geth' and image 'ethpandaops/lighthouse:stable'
There was an error executing Starlark code 
An error occurred executing instruction (number 30) at github.com/ethpandaops/ethereum-package/src/cl/lighthouse/lighthouse_launcher.star[161:38]:
  add_service(name="cl-1-lighthouse-geth", config=ServiceConfig(image="ethpandaops/lighthouse:stable", ports={"http": PortSpec(number=4000, transport_protocol="TCP", application_protocol="http"), "metrics": PortSpec(number=5054, transport_protocol="TCP", application_protocol="http"), "tcp-discovery": PortSpec(number=9000, transport_protocol="TCP", application_protocol=""), "udp-discovery": PortSpec(number=9000, transport_protocol="UDP", application_protocol="")}, public_ports={}, files={"/jwt": "jwt_file", "/network-configs": "el_cl_genesis_data"}, cmd=["lighthouse", "beacon_node", "--debug-level=info", "--datadir=/data/lighthouse/beacon-data", "--disable-enr-auto-update", "--enr-address=KURTOSIS_IP_ADDR_PLACEHOLDER", "--enr-udp-port=9000", "--enr-tcp-port=9000", "--listen-address=0.0.0.0", "--port=9000", "--http", "--http-address=0.0.0.0", "--http-port=4000", "--http-allow-sync-stalled", "--slots-per-restore-point=32", "--disable-packet-filter", "--execution-endpoints=http://{{kurtosis:8db3f51e913e4359817782acfe6d80e8:ip_address.runtime_value}}:8551", "--jwt-secrets=/jwt/jwtsecret", "--suggested-fee-recipient=0x8943545177806ED17B9F23F0a21ee5948eCaa776", "--subscribe-all-subnets", "--metrics", "--metrics-address=0.0.0.0", "--metrics-allow-origin=*", "--metrics-port=5054", "--enable-private-discovery", "--testnet-dir=/network-configs"], env_vars={"RUST_BACKTRACE": "full"}, private_ip_address_placeholder="KURTOSIS_IP_ADDR_PLACEHOLDER", max_cpu=1000, min_cpu=50, max_memory=1024, min_memory=256, ready_conditions=ReadyCondition(recipe=GetHttpRequestRecipe(port_id="http", endpoint="/eth/v1/node/health"), field="code", assertion="IN", target_value=[200, 206], timeout="15m"), labels={"ethereum-package.client": "lighthouse", "ethereum-package.client-image": "ethpandaops-lighthouse_stable", "ethereum-package.client-type": "beacon", "ethereum-package.connected-client": "geth", "ethereum-package.sha256": ""}, tolerations=[], node_selectors={}))
  Caused by: Unexpected error occurred starting service 'cl-1-lighthouse-geth'
  Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'cl-1-lighthouse-geth' with private IP '172.16.0.13'; this is usually due to a misconfiguration in the service itself, so here are the logs:
  == SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Jun 26 23:38:20.557 INFO Logging to file                         path: "/data/lighthouse/beacon-data/beacon/logs/beacon.log"
  Jun 26 23:38:20.573 INFO Lighthouse started                      version: Lighthouse/v5.2.0-f1d88ba
  Jun 26 23:38:20.573 INFO Configured for network                  name: custom (/network-configs)
  Jun 26 23:38:20.574 INFO Data directory initialised              datadir: /data/lighthouse/beacon-data
  Jun 26 23:38:20.575 WARN Discv5 packet filter is disabled
  Jun 26 23:38:20.575 WARN Ignoring --http-allow-sync-stalled      info: this flag is deprecated and will be removed
  Jun 26 23:38:20.583 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
  Jun 26 23:38:20.615 INFO Blob DB initialized                     oldest_blob_slot: Some(Slot(0)), path: "/data/lighthouse/beacon-data/beacon/blobs_db", service: freezer_db
  Jun 26 23:38:20.647 INFO Starting from known genesis state       service: beacon

  == FINISHED SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.0.13' and port spec '{privatePortSpec:0xc000d9cb10}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.0.13:9000' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.0.13:9000: i/o timeout

Error encountered running Starlark code.

⭐ us on GitHub - https://github.com/kurtosis-tech/kurtosis
INFO[2024-06-27T07:40:20+08:00] =================================================== 
INFO[2024-06-27T07:40:20+08:00] ||          Created enclave: dry-geyser          || 
INFO[2024-06-27T07:40:20+08:00] =================================================== 
Name:            dry-geyser
UUID:            8a9886d10290
Status:          RUNNING
Creation Time:   Thu, 27 Jun 2024 07:36:57 CST
Flags:           

========================================= Files Artifacts =========================================
UUID           Name
bb406afedb3a   1-lighthouse-geth-0-63-0
0c1bef28eb78   2-lighthouse-geth-64-127-0
a86050e79779   3-teku-geth-128-191-0
357236e5d3f6   el_cl_genesis_data
56a5a841d366   final-genesis-timestamp
6e53c4476eb2   genesis-el-cl-env-file
2c691f370de9   genesis_validators_root
bda9564aa987   jwt_file
e7799dd9ca68   keymanager_file
36c3d12fe1aa   prysm-password

========================================== User Services ==========================================
UUID           Name                                             Ports                                         Status
294aacfef506   el-1-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49326       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49325          
                                                                rpc: 8545/tcp -> http://127.0.0.1:49328       
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49324   
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49187   
                                                                ws: 8546/tcp -> 127.0.0.1:49327               
21e3516db10e   el-2-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49331       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49330          
                                                                rpc: 8545/tcp -> http://127.0.0.1:49333       
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49329   
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49188   
                                                                ws: 8546/tcp -> 127.0.0.1:49332               
1392da84cc1e   el-3-geth-teku                                   engine-rpc: 8551/tcp -> 127.0.0.1:49336       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49335          
                                                                rpc: 8545/tcp -> http://127.0.0.1:49338       
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49334   
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49189   
                                                                ws: 8546/tcp -> 127.0.0.1:49337               
154ff54a408d   validator-key-generation-cl-validator-keystore   <none>                                        RUNNING

(base) root@ST-ZY-28:/var/zk_evm# 
INFO[2024-06-27T07:40:20+08:00] ===================================================
Name:            dry-geyser
UUID:            8a9886d10290
Status:          RUNNING
Creation Time:   Thu, 27 Jun 2024 07:36:57 CST
Flags:

========================================= Files Artifacts =========================================
UUID           Name
bb406afedb3a   1-lighthouse-geth-0-63-0
0c1bef28eb78   2-lighthouse-geth-64-127-0
a86050e79779   3-teku-geth-128-191-0
357236e5d3f6   el_cl_genesis_data
56a5a841d366   final-genesis-timestamp
6e53c4476eb2   genesis-el-cl-env-file
2c691f370de9   genesis_validators_root
bda9564aa987   jwt_file
e7799dd9ca68   keymanager_file
36c3d12fe1aa   prysm-password

========================================== User Services ==========================================
UUID           Name                                             Ports                                         Status
294aacfef506   el-1-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49326       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49325
                                                                rpc: 8545/tcp -> http://127.0.0.1:49328
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49324
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49187
                                                                ws: 8546/tcp -> 127.0.0.1:49327
21e3516db10e   el-2-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49331       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49330
                                                                rpc: 8545/tcp -> http://127.0.0.1:49333
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49329
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49188
                                                                ws: 8546/tcp -> 127.0.0.1:49332
1392da84cc1e   el-3-geth-teku                                   engine-rpc: 8551/tcp -> 127.0.0.1:49336       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49335
                                                                rpc: 8545/tcp -> http://127.0.0.1:49338
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49334
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49189
                                                                ws: 8546/tcp -> 127.0.0.1:49337
154ff54a408d   validator-key-generation-cl-validator-keystore   <none>                                        RUNNING

(base) root@ST-ZY-28:/var/zk_evm# 

  == FINISHED SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.0.13' and port spec '{privatePortSpec:0xc000d9cb10}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.0.13:9000' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.0.13:9000: i/o timeout

Error encountered running Starlark code.

⭐ us on GitHub - https://github.com/kurtosis-tech/kurtosis
INFO[2024-06-27T07:40:20+08:00] ===================================================
INFO[2024-06-27T07:40:20+08:00] ||          Created enclave: dry-geyser          ||
INFO[2024-06-27T07:40:20+08:00] ===================================================
Name:            dry-geyser
UUID:            8a9886d10290
Status:          RUNNING
Creation Time:   Thu, 27 Jun 2024 07:36:57 CST
Flags:

========================================= Files Artifacts =========================================
UUID           Name
bb406afedb3a   1-lighthouse-geth-0-63-0
0c1bef28eb78   2-lighthouse-geth-64-127-0
a86050e79779   3-teku-geth-128-191-0
357236e5d3f6   el_cl_genesis_data
56a5a841d366   final-genesis-timestamp
6e53c4476eb2   genesis-el-cl-env-file
2c691f370de9   genesis_validators_root
bda9564aa987   jwt_file
e7799dd9ca68   keymanager_file
36c3d12fe1aa   prysm-password

========================================== User Services ==========================================
UUID           Name                                             Ports                                         Status
294aacfef506   el-1-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49326       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49325
                                                                rpc: 8545/tcp -> http://127.0.0.1:49328
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49324
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49187
                                                                ws: 8546/tcp -> 127.0.0.1:49327
21e3516db10e   el-2-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49331       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49330
                                                                rpc: 8545/tcp -> http://127.0.0.1:49333
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49329
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49188
                                                                ws: 8546/tcp -> 127.0.0.1:49332
1392da84cc1e   el-3-geth-teku                                   engine-rpc: 8551/tcp -> 127.0.0.1:49336       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49335
                                                                rpc: 8545/tcp -> http://127.0.0.1:49338
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49334
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49189
                                                                ws: 8546/tcp -> 127.0.0.1:49337
154ff54a408d   validator-key-generation-cl-validator-keystore   <none>                                        RUNNING

my network_params2.yaml is：

participants:
  - el_type: geth
    cl_type: lighthouse
    count: 2
  - el_type: geth
    cl_type: teku
network_params:
  network_id: "585858"
additional_services:
  - dora

port is free

`

(base) root@ST-ZY-28:/var/zk_evm# sudo netstat -tuln | grep -E '9000|4000|5054' (base) root@ST-ZY-28:/var/zk_evm# sudo ufw status sudo ufw allow 4000/tcp sudo ufw allow 5054/tcp sudo ufw allow 9000/tcp sudo ufw allow 9000/udp Status: inactive (base) root@ST-ZY-28:/var/zk_evm#

`

barnabasbusa commented 1 week ago

Can you try to update your docker engine version to 27.0.2. And possibly try to bump your ubuntu to 24.04?

KatyaRyazantseva commented 2 days ago

I have the same issue but on Mac. The latest docker engine version for mac is v.26.1.4. Do I have any chance?

MacOS: Monterey 12.7.5 Docker Desktop: 4.31.0 Docker Engine: 26.1.4 Kurtosis CLI: 0.90.1

barnabasbusa commented 2 days ago

Could you check if you seeing the same issues with a different cl_type too?

e.g lodestar ?

KatyaRyazantseva commented 2 days ago

I've checked lodestar, prysm, teku, nimbus, grandine. The issue occurs only for lighthouse. And I have to comment on port_publisher param. With default settings it doesn't work at all.

# port_publisher:
#   public_port_start: null
#   nat_exit_ip: KURTOSIS_IP_ADDR_PLACEHOLDER