0xPolygon / kurtosis-cdk

A Kurtosis package that deploys a private, portable, and modular Polygon CDK devnet
https://docs.polygon.technology/cdk
Apache License 2.0
41 stars 72 forks source link

error calling network address '172.16.0.9:9091' with port protocol #169

Closed izzi12 closed 1 month ago

izzi12 commented 3 months ago

System information

VM Virtualbox Ubuntu (64-bit)

Commit id

c139e0c

Tools versions

✅ kurtosis 0.89.0 is installed, meets the requirement (=0.89). ✅ docker 24.0.7 is installed, meets the requirement (>=24.7).

You might as well need the following tools to interact with the environment... ✅ jq 1.7 is installed. ❌ yq 0.0.0 is installed, but only version 3.2 is supported by the package.

Description & steps to reproduce

The local L1 setup is working fine but when I try for both Sepolia and Holesky testnets I get the following error in "Adding service with name 'zkevm-node-synchronizer-001' and image '0xpolygon/cdk-validium-node:0.6.5-cdk'"

Adding service with name 'zkevm-node-synchronizer-001' and image '0xpolygon/cdk-validium-node:0.6.5-cdk'
There was an error executing Starlark code 
An error occurred executing instruction (number 26) at github.com/0xPolygon/kurtosis-cdk/lib/zkevm_node.star[52:21]:
  add_service(name="zkevm-node-synchronizer-001", config=ServiceConfig(image="0xpolygon/cdk-validium-node:0.6.5-cdk", ports={"pprof": PortSpec(number=6060, application_protocol="http"), "prometheus": PortSpec(number=9091, application_protocol="http")}, files={"/etc/zkevm": Directory(artifact_names=["trusted-node-config", "genesis"])}, entrypoint=["/app/zkevm-node"], cmd=["run", "--cfg=/etc/zkevm/node-config.toml", "--network=custom", "--custom-network-file=/etc/zkevm/genesis.json", "--components=synchronizer"]))
  Caused by: Unexpected error occurred starting service 'zkevm-node-synchronizer-001'
  Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'zkevm-node-synchronizer-001' with private IP '172.16.0.9'; this is usually due to a misconfiguration in the service itself, so here are the logs:
  == SERVICE 'zkevm-node-synchronizer-001' LOGS ===================================
  panic: failed to load genesis configuration from file. Error: json: cannot unmarshal non-string into Go struct field L1Config.L1Config.polygonZkEVMGlobalExitRootAddress of type common.Address

  goroutine 1 [running]:
  github.com/0xPolygonHermez/zkevm-node/config.(*Config).loadNetworkConfig(0xc000585b00, 0x15ecc40?)
    /home/runner/work/cdk-validium-node/cdk-validium-node/config/network.go:74 +0x1d3
  github.com/0xPolygonHermez/zkevm-node/config.Load(0xc00017e390?, 0x1)
    /home/runner/work/cdk-validium-node/cdk-validium-node/config/config.go:186 +0x432
  main.start(0xc0001de880)
    /home/runner/work/cdk-validium-node/cdk-validium-node/cmd/run.go:52 +0x37
  github.com/urfave/cli/v2.(*Command).Run(0xc000137760, 0xc0001de880, {0xc0005b5680, 0x5, 0x5})
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/command.go:277 +0x9d8
  github.com/urfave/cli/v2.(*Command).Run(0xc000460420, 0xc0001de740, {0xc0000d4120, 0x6, 0x6})
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/command.go:270 +0xc25
  github.com/urfave/cli/v2.(*App).RunContext(0xc000590d20, {0x1d5c320?, 0x2b2b440}, {0xc0000d4120, 0x6, 0x6})
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/app.go:335 +0x5db
  github.com/urfave/cli/v2.(*App).Run(...)
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/app.go:309
  main.main()
    /home/runner/work/cdk-validium-node/cdk-validium-node/cmd/main.go:198 +0xd3b

  == FINISHED SERVICE 'zkevm-node-synchronizer-001' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.0.9' and port spec '{privatePortSpec:0xc002bc0b70}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.0.9:9091' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.0.9:9091: connect: no route to host

Error encountered running Starlark code.

Desired behavior

The deployment on Sepolia should run smoothly.

What is the severity of this bug?

Critical; I am blocked and Kurtosis CDK is unusable for me because of this bug.

izzi12 commented 3 months ago

I ran the setup in a new environment and got a similar error but logs here are slightly different. There was an error executing Starlark code An error occurred executing instruction (number 26) at github.com/0xPolygon/kurtosis-cdk/lib/zkevm_node.star[52:21]: add_service(name="zkevm-node-synchronizer-001", config=ServiceConfig(image="0xpolygon/cdk-validium-node:0.6.5-cdk", ports={"pprof": PortSpec(number=6060, application_protocol="http"), "prometheus": PortSpec(number=9091, application_protocol="http")}, files={"/etc/zkevm": Directory(artifact_names=["trusted-node-config", "genesis"])}, entrypoint=["/app/zkevm-node"], cmd=["run", "--cfg=/etc/zkevm/node-config.toml", "--network=custom", "--custom-network-file=/etc/zkevm/genesis.json", "--components=synchronizer"])) Caused by: Unexpected error occurred starting service 'zkevm-node-synchronizer-001' Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'zkevm-node-synchronizer-001' with private IP '172.16.0.9'; this is usually due to a misconfiguration in the service itself, so here are the logs: == SERVICE 'zkevm-node-synchronizer-001' LOGS =================================== 2024-07-02T15:43:46.732Z INFO config/config.go:169 error reading config file: %!(EXTRA viper.ConfigParseError=While parsing config: toml: basic strings cannot have new lines) {"pid": 7, "version": "0.6.5+cdk"} 2024-07-02T15:43:46.732Z FATAL cmd/main.go:200 While parsing config: toml: basic strings cannot have new lines /home/runner/work/cdk-validium-node/cdk-validium-node/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs() /home/runner/work/cdk-validium-node/cdk-validium-node/log/log.go:223 github.com/0xPolygonHermez/zkevm-node/log.Fatal() /home/runner/work/cdk-validium-node/cdk-validium-node/cmd/main.go:200 main.main() /opt/hostedtoolcache/go/1.21.9/x64/src/runtime/proc.go:267 runtime.main() {"pid": 7, "version": "0.6.5+cdk"} main.main /home/runner/work/cdk-validium-node/cdk-validium-node/cmd/main.go:200 runtime.main /opt/hostedtoolcache/go/1.21.9/x64/src/runtime/proc.go:267

== FINISHED SERVICE 'zkevm-node-synchronizer-001' LOGS =================================== Caused by: An error occurred while waiting for all TCP and UDP ports to be open Caused by: Unsuccessful ports check for IP '172.16.0.9' and port spec '{privatePortSpec:0xc0006be2d0}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached Caused by: An error occurred while calling network address '172.16.0.9:6060' with port protocol 'TCP' and using time out '200ms' Caused by: dial tcp 172.16.0.9:6060: i/o timeout

Error encountered running Starlark code.

izzi12 commented 3 months ago

In between i also got this error, Not sure if that's the reason...

[2024-07-03 18:05:30] Combining contract deploy files cp: cannot stat '/opt/zkevm-contracts/deployment/v2/create_rollup_output.json': No such file or directory /opt /opt/zkevm /opt [2024-07-03 18:05:30] Creating combined.json jq: Bad JSON in --slurpfile rollup create_rollup_output.json: Could not open create_rollup_output.json: No such file or directory Transformation complete. Output written to dynamic-kurtosis-allocs.json jq: error (at dynamic-kurtosis-conf.json:6): Expected JSON value (while parsing '') [2024-07-03 18:05:32] Approving the rollup address to transfer POL tokens on behalf of the sequencer Error: Failed to resolve ENS name: ens name not found: [2024-07-03 18:05:33] Setting the data availability committee Error: Failed to resolve ENS name: ens name not found: [2024-07-03 18:05:35] Setting the data availability protocol Error: Failed to resolve ENS name: ens name not found: [2024-07-03 18:05:36] Granting the aggregator role to the agglayer so that it can also verify batches Error: Failed to resolve ENS name: ens name not found:


Creating keystores for zkevm-node/cdk-validium components Command returned with exit code '0' with no output

leovct commented 3 months ago

Hi @izzi12, can you take a look at this comment and see if it resolves your issue?

the-first-elder commented 3 months ago

Currently having this error as well and the network address issue

I have yq intsalled

In between i also got this error, Not sure if that's the reason...

[2024-07-03 18:05:30] Combining contract deploy files cp: cannot stat '/opt/zkevm-contracts/deployment/v2/create_rollup_output.json': No such file or directory /opt /opt/zkevm /opt [2024-07-03 18:05:30] Creating combined.json jq: Bad JSON in --slurpfile rollup create_rollup_output.json: Could not open create_rollup_output.json: No such file or directory Transformation complete. Output written to dynamic-kurtosis-allocs.json jq: error (at dynamic-kurtosis-conf.json:6): Expected JSON value (while parsing '') [2024-07-03 18:05:32] Approving the rollup address to transfer POL tokens on behalf of the sequencer Error: Failed to resolve ENS name: ens name not found: [2024-07-03 18:05:33] Setting the data availability committee Error: Failed to resolve ENS name: ens name not found: [2024-07-03 18:05:35] Setting the data availability protocol Error: Failed to resolve ENS name: ens name not found: [2024-07-03 18:05:36] Granting the aggregator role to the agglayer so that it can also verify batches Error: Failed to resolve ENS name: ens name not found:

Creating keystores for zkevm-node/cdk-validium components Command returned with exit code '0' with no output

izzi12 commented 3 months ago

@leovct - I have made the changes in the comment, but this issue is unresolved. I have created a fork of the project here with all my settings. Please have a look. izzi-kurtosis-cdk

jhkimqd commented 2 months ago

@izzi12 @the-first-elder first, could you double check that the previous enclave is cleaned with kurtosis clean --all, and retry the Sepolia deployments with a new salt?

the-first-elder commented 2 months ago

@izzi12 @the-first-elder first, could you double check that the previous enclave is cleaned with kurtosis clean --all, and retry the Sepolia deployments with a new salt?

Made the recommended changes,had same error

izzi12 commented 2 months ago

@izzi12 @the-first-elder first, could you double check that the previous enclave is cleaned with kurtosis clean --all, and retry the Sepolia deployments with a new salt?

Same here. same error.

rebelArtists commented 2 months ago

ack, will attempt to recreate and keep the group posted

izzi12 commented 2 months ago

ack, will attempt to recreate and keep the group posted Please use my git repository. https://github.com/izzi12/kurtosis-cdk/

Pana commented 2 months ago

Same issue encountered

Pana commented 2 months ago

@the-first-elder your issue is not related to this one, should be caused by the contract deployment step failed

the-first-elder commented 2 months ago

@the-first-elder your issue is not related to this one, should be caused by the contract deployment step failed

how do i fix it ser

Pana commented 2 months ago

You can try to deploy https://github.com/0xPolygonhermez/zkevm-contracts/ on your chain first, this will be helpful, if you can succeed this, then you can come back to solve cdk issue

Pana commented 2 months ago

@arnaubennassar can you help us solve this issue, seems this is a kurtosis or docker related issue, right ?

Pana commented 2 months ago

@jarrodwatts @rebelArtists can you help check this issue ?

izzi12 commented 2 months ago

@jarrodwatts - I followed your Sepolia testnet video precisely, but I'm really stuck now.

Pana commented 2 months ago

After learning kurtosis, figure out this error is meaning kurtosis check zkevm-node-synchronizer-001's pprof and promethus port failed. But don't know why zkevm-node-synchronizer-001 service can not started.

Pana commented 2 months ago

The Polygon CDK is really hard to use and not mature. I think the team want us contact the official or IP to pay for it

izzi12 commented 2 months ago

Sadly, I think you are right. :( I did the same setup on my MAC insted of Linux server and got a similar error. Check this out...

Adding service with name 'zkevm-node-synchronizer-001' and image '0xpolygon/cdk-validium-node:0.6.5-cdk' There was an error executing Starlark code An error occurred executing instruction (number 26) at github.com/0xPolygon/kurtosis-cdk/lib/zkevm_node.star[52:21]: add_service(name="zkevm-node-synchronizer-001", config=ServiceConfig(image="0xpolygon/cdk-validium-node:0.6.5-cdk", ports={"pprof": PortSpec(number=6060, application_protocol="http"), "prometheus": PortSpec(number=9091, application_protocol="http")}, files={"/etc/zkevm": Directory(artifact_names=["trusted-node-config", "genesis"])}, entrypoint=["/app/zkevm-node"], cmd=["run", "--cfg=/etc/zkevm/node-config.toml", "--network=custom", "--custom-network-file=/etc/zkevm/genesis.json", "--components=synchronizer"])) Caused by: Unexpected error occurred starting service 'zkevm-node-synchronizer-001' Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'zkevm-node-synchronizer-001' with private IP '172.16.0.9'; this is usually due to a misconfiguration in the service itself, so here are the logs: == SERVICE 'zkevm-node-synchronizer-001' LOGS =================================== panic: failed to load genesis configuration from file. Error: json: cannot unmarshal non-string into Go struct field L1Config.L1Config.polygonZkEVMGlobalExitRootAddress of type common.Address

goroutine 1 [running]: github.com/0xPolygonHermez/zkevm-node/config.(Config).loadNetworkConfig(0xc00012db00, 0x15ecc40?) /home/runner/work/cdk-validium-node/cdk-validium-node/config/network.go:74 +0x1d3 github.com/0xPolygonHermez/zkevm-node/config.Load(0xc00040fc80?, 0x1) /home/runner/work/cdk-validium-node/cdk-validium-node/config/config.go:186 +0x432 main.start(0xc0005a8340) /home/runner/work/cdk-validium-node/cdk-validium-node/cmd/run.go:52 +0x37 github.com/urfave/cli/v2.(Command).Run(0xc0001a4dc0, 0xc0005a8340, {0xc0000ae280, 0x5, 0x5}) /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/command.go:277 +0x9d8 github.com/urfave/cli/v2.(Command).Run(0xc000718160, 0xc0005a8200, {0xc000142120, 0x6, 0x6}) /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/command.go:270 +0xc25 github.com/urfave/cli/v2.(App).RunContext(0xc0000b45a0, {0x1d5c320?, 0x2b2b440}, {0xc000142120, 0x6, 0x6}) /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/app.go:335 +0x5db github.com/urfave/cli/v2.(*App).Run(...) /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.26.0/app.go:309 main.main() /home/runner/work/cdk-validium-node/cdk-validium-node/cmd/main.go:198 +0xd3b

== FINISHED SERVICE 'zkevm-node-synchronizer-001' LOGS =================================== Caused by: An error occurred while waiting for all TCP and UDP ports to be open Caused by: Unsuccessful ports check for IP '172.16.0.9' and port spec '{privatePortSpec:0xc0028769c0}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached Caused by: An error occurred while calling network address '172.16.0.9:6060' with port protocol 'TCP' and using time out '200ms' Caused by: dial tcp 172.16.0.9:6060: i/o timeout

Error encountered running Starlark code.

izzi12 commented 1 month ago

I tried to debug the error and it seems node-config.toml file is giving the error 2024-08-06 21_29_11-TOML Lint - The TOML Validator Any idea how we can fix this.

izzi12 commented 1 month ago

The error originates from Deploying PolygonZKEVMDeployer.

[2024-08-07 08:48:49] Step 3: Deploying PolygonZKEVMDeployer

@0xpolygonhermez/zkevm-contracts@3.0.0 npx hardhat run deployment/v2/2_deployPolygonZKEVMDeployer.ts --network localhost

ProviderError: only replay-protected (eip-155) transactions allowed over rpc at HttpProvider.request (/opt/zkevm-contracts/node_modules/hardhat/src/internal/core/providers/http.ts:88:21) at processTicksAndRejections (node:internal/process/task_queues:95:5) at async Promise.all (index 0) at async HardhatEthersProvider.broadcastTransaction (/opt/zkevm-contracts/node_modules/@nomicfoundation/hardhat-ethers/src/internal/hardhat-ethers-provider.ts:266:33) at async deployPolygonZkEVMDeployer (/opt/zkevm-contracts/deployment/helpers/deployment-helpers.ts:71:12) at async main (/opt/zkevm-contracts/deployment/v2/2_deployPolygonZKEVMDeployer.ts:73:54) [2024-08-07 08:49:04] Step 4: Deploying contracts

@0xpolygonhermez/zkevm-contracts@3.0.0 npx hardhat run deployment/v2/3_deployContracts.ts --network localhost

Error: Missing parameter: zkEVMDeployerAddress at main (/opt/zkevm-contracts/deployment/v2/3_deployContracts.ts:86:19) at Object. (/opt/zkevm-contracts/deployment/v2/3_deployContracts.ts:552:1) at Module._compile (node:internal/modules/cjs/loader:1358:14) at Module.m._compile (/opt/zkevm-contracts/node_modules/ts-node/src/index.ts:1618:23) at Module._extensions..js (node:internal/modules/cjs/loader:1416:10) at Object.require.extensions. [as .ts] (/opt/zkevm-contracts/node_modules/ts-node/src/index.ts:1621:12) at Module.load (node:internal/modules/cjs/loader:1208:32) at Function.Module._load (node:internal/modules/cjs/loader:1024:12) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:174:12) at node:internal/main/run_main_module:28:49 [2024-08-07 08:49:05] Step 5: Creating rollup

@0xpolygonhermez/zkevm-contracts@3.0.0 npx hardhat run deployment/v2/4_createRollup.ts --network localhost

Error: Cannot find module './deploy_output.json' Require stack:


Please look into this.

Pana commented 1 month ago

PolygonZKEVMDeployer is deployed using keyless method, and your network do not support pre-155 tx, it said "only replay-protected (eip-155) transactions allowed over rpc" so it failed. you need fix this first

Pana commented 1 month ago

I tried to debug the error and it seems node-config.toml file is giving the error 2024-08-06 21_29_11-TOML Lint - The TOML Validator Any idea how we can fix this.

this is a toml template, kurtosis will render real value, when deploying service , this is not a issue

izzi12 commented 1 month ago

PolygonZKEVMDeployer is deployed using keyless method, and your network do not support pre-155 tx, it said "only replay-protected (eip-155) transactions allowed over rpc" so it failed. you need fix this first

How can I fix this?

leovct commented 1 month ago

Closing this issue and tracking issues related to Sepolia deployment here: #238