barnabasbusa
commented
4 months ago Can you try to update your docker engine version to 27.0.2. And possibly try to bump your ubuntu to 24.04?
Open davidLeeeeeeeeeeee opened 4 months ago
barnabasbusa
commented
4 months ago Can you try to update your docker engine version to 27.0.2. And possibly try to bump your ubuntu to 24.04?
KatyaRyazantseva
commented
4 months ago I have the same issue but on Mac. The latest docker engine version for mac is v.26.1.4. Do I have any chance?
MacOS: Monterey 12.7.5 Docker Desktop: 4.31.0 Docker Engine: 26.1.4 Kurtosis CLI: 0.90.1
barnabasbusa
commented
4 months ago Could you check if you seeing the same issues with a different cl_type too?
e.g lodestar ?
KatyaRyazantseva
commented
4 months ago I've checked lodestar, prysm, teku, nimbus, grandine. The issue occurs only for lighthouse. And I have to comment on port_publisher param. With default settings it doesn't work at all.
# port_publisher:
# public_port_start: null
# nat_exit_ip: KURTOSIS_IP_ADDR_PLACEHOLDERThe port publisher parameters have been reworked in https://github.com/ethpandaops/ethereum-package/pull/677.
Can you ensure that your virtual disk has enough free space and that you running the latest of everything? Kurtosis version 1.0.0 or newer, and the main branch of this repo.
KatyaRyazantseva
commented
4 months ago I have 30Gb available, Kurtosis version 1.0.0 and I am running kurtosis run github.com/ethpandaops/ethereum-package. I guess it's the main brunch. I deleted all the lighthouse docker images before running. Currently I have ethpandaops/lighthouse:stable (created 14 days ago).
Here is the latest error. The port can change from launch to launch: 9000, 4000, 5054.
Adding service with name 'cl-1-lighthouse-geth' and image 'ethpandaops/lighthouse:stable'
There was an error executing Starlark code
An error occurred executing instruction (number 25) at github.com/ethpandaops/ethereum-package/src/cl/lighthouse/lighthouse_launcher.star[135:38]:
add_service(name="cl-1-lighthouse-geth", config=ServiceConfig(image="ethpandaops/lighthouse:stable", ports={"http": PortSpec(number=4000, transport_protocol="TCP", application_protocol="http"), "metrics": PortSpec(number=5054, transport_protocol="TCP", application_protocol="http"), "tcp-discovery": PortSpec(number=9000, transport_protocol="TCP", application_protocol=""), "udp-discovery": PortSpec(number=9000, transport_protocol="UDP", application_protocol="")}, public_ports={}, files={"/jwt": "jwt_file", "/network-configs": "el_cl_genesis_data"}, cmd=["lighthouse", "beacon_node", "--debug-level=info", "--datadir=/data/lighthouse/beacon-data", "--disable-enr-auto-update", "--enr-address=KURTOSIS_IP_ADDR_PLACEHOLDER", "--enr-udp-port=9000", "--enr-tcp-port=9000", "--listen-address=0.0.0.0", "--port=9000", "--http", "--http-address=0.0.0.0", "--http-port=4000", "--http-allow-sync-stalled", "--slots-per-restore-point=32", "--disable-packet-filter", "--execution-endpoints=http://{{kurtosis:a77cdf0a62b648ffafc7b284ff43b088:ip_address.runtime_value}}:8551", "--jwt-secrets=/jwt/jwtsecret", "--suggested-fee-recipient=0x8943545177806ED17B9F23F0a21ee5948eCaa776", "--subscribe-all-subnets", "--metrics", "--metrics-address=0.0.0.0", "--metrics-allow-origin=*", "--metrics-port=5054", "--enable-private-discovery", "--testnet-dir=/network-configs"], env_vars={"RUST_BACKTRACE": "full"}, private_ip_address_placeholder="KURTOSIS_IP_ADDR_PLACEHOLDER", max_cpu=1000, min_cpu=50, max_memory=1024, min_memory=256, ready_conditions=ReadyCondition(recipe=GetHttpRequestRecipe(port_id="http", endpoint="/eth/v1/node/health"), field="code", assertion="IN", target_value=[200, 206], timeout="15m"), labels={"ethereum-package.client": "lighthouse", "ethereum-package.client-image": "ethpandaops-lighthouse_stable", "ethereum-package.client-type": "beacon", "ethereum-package.connected-client": "geth", "ethereum-package.sha256": ""}, tolerations=[], node_selectors={}))
Caused by: Unexpected error occurred starting service 'cl-1-lighthouse-geth'
Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'cl-1-lighthouse-geth' with private IP '172.16.4.11'; this is usually due to a misconfiguration in the service itself, so here are the logs:
== SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
Jul 11 16:06:50.076 INFO Logging to file path: "/data/lighthouse/beacon-data/beacon/logs/beacon.log"
Jul 11 16:06:50.221 INFO Lighthouse started version: Lighthouse/v5.2.0-f1d88ba
Jul 11 16:06:50.222 INFO Configured for network name: custom (/network-configs)
Jul 11 16:06:50.244 INFO Data directory initialised datadir: /data/lighthouse/beacon-data
Jul 11 16:06:50.247 WARN Discv5 packet filter is disabled
Jul 11 16:06:50.247 WARN Ignoring --http-allow-sync-stalled info: this flag is deprecated and will be removed
Jul 11 16:06:50.259 INFO Deposit contract address: 0x4242424242424242424242424242424242424242, deploy_block: 0
Jul 11 16:06:50.309 INFO Blob DB initialized oldest_blob_slot: Some(Slot(0)), path: "/data/lighthouse/beacon-data/beacon/blobs_db", service: freezer_db
Jul 11 16:06:50.349 INFO Starting from known genesis state service: beacon
== FINISHED SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
Caused by: An error occurred while waiting for all TCP and UDP ports to be open
Caused by: Unsuccessful ports check for IP '172.16.4.11' and port spec '{privatePortSpec:0xc00070db90}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
Caused by: An error occurred while calling network address '172.16.4.11:9000' with port protocol 'TCP' and using time out '200ms'
Caused by: dial tcp 172.16.4.11:9000: i/o timeout
Error encountered running Starlark code.can you try to do cl_max_mem override to something more than 1024? Let's try with 4096.
KatyaRyazantseva
commented
4 months ago What I found out, if I run kurtosis run github.com/ethpandaops/ethereum-package, I have this timeout error. But if I run the same with --args-file network_params.yaml, it works fine. Even though the remote file and the local file are identical. So, even not applying 4096 it works. Checked several times, it is reproduced every time.
Ubuntu 22.04 LTS
Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3
kurtosis-cli version (0.90.1)
my network_params2.yaml is：
port is free
`
(base) root@ST-ZY-28:/var/zk_evm# sudo netstat -tuln | grep -E '9000|4000|5054' (base) root@ST-ZY-28:/var/zk_evm# sudo ufw status sudo ufw allow 4000/tcp sudo ufw allow 5054/tcp sudo ufw allow 9000/tcp sudo ufw allow 9000/udp Status: inactive (base) root@ST-ZY-28:/var/zk_evm#
`