search

zkSync-Community-Hub / zksync-developers

Project for Discussions with the ZKsync Community.
369 stars 233 forks source link

node stops while syncing - external-node-1 exited with code 137 #527

Closed shaky closed 6 months ago

shaky commented 6 months ago

hi there,

I'm trying to run the node announced here https://github.com/zkSync-Community-Hub/zksync-developers/discussions/526 but seems like it stops running after approximately 10 mins.

docker compose --file mainnet-external-node-docker-compose.yml up

WARN[0000] /Users/censored/Projects/zksync-era/docs/guides/external-node/docker-compose-examples/mainnet-external-node-docker-compose.yml: `version` is obsolete 
[+] Running 4/0
 ✔ Container docker-compose-examples-postgres-1       Running                                                                                                                 0.0s 
 ✔ Container docker-compose-examples-prometheus-1     Running                                                                                                                 0.0s 
 ✔ Container docker-compose-examples-grafana-1        Running                                                                                                                 0.0s 
 ✔ Container docker-compose-examples-external-node-1  Created                                                                                                                 0.0s 
Attaching to external-node-1, grafana-1, postgres-1, prometheus-1
external-node-1  | 2024-05-13T18:10:04.413731Z  INFO zksync_external_node: No sentry URL was provided
external-node-1  | 2024-05-13T18:10:05.657965Z  INFO zksync_external_node::metrics: Metadata for rustc that this EN was compiled with: RustcMetadata { version: "1.74.0-nightly", commit_hash: Some("5c6a7e71cd66705c31c9af94077901a220f0870c"), commit_date: Some("2023-08-20"), channel: "nightly", host: "aarch64-unknown-linux-gnu", llvm: Some("17.0") }
external-node-1  | 2024-05-13T18:10:05.658057Z  INFO zksync_external_node::metrics: Setting general node information: ExternalNodeInfo { server_version: "23.0.0", l1_chain_id: 1, l2_chain_id: 324, postgres_pool_size: 10 }
external-node-1  | 2024-05-13T18:10:05.681284Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { max_size: 10, acquire_timeout: 30s, statement_timeout: None }
external-node-1  | 2024-05-13T18:10:05.681419Z  INFO zksync_external_node: Main node URL is: https://zksync2-mainnet.zksync.io:443/
external-node-1  | 2024-05-13T18:10:05.682213Z  WARN zksync_external_node: The external node is in the alpha phase, and should be used with caution.
external-node-1  | 2024-05-13T18:10:05.682235Z  INFO zksync_external_node: Started the external node
external-node-1  | 2024-05-13T18:10:05.697082Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { max_size: 1, acquire_timeout: 30s, statement_timeout: None }
external-node-1  | 2024-05-13T18:10:05.697245Z  INFO zksync_external_node::version_sync_task: Starting syncing protocol version of blocks
external-node-1  | 2024-05-13T18:10:05.709619Z  INFO zksync_external_node::init: Node has no genesis L1 batch and snapshot recovery information: SnapshotRecoveryStatus { l1_batch_number: L1BatchNumber(480164), l1_batch_root_hash: 0xccf9d6c8fd8dbf965da4e366b69bf299820c1c469ddc4090af76a77cebd1ec87, l1_batch_timestamp: 1715556481, miniblock_number: MiniblockNumber(33835879), miniblock_hash: 0x0ca9fc8ca1c6e7834a9aee185686f1cee9689aa1be5f080cf4d6b4e6f3e6c8fc, miniblock_timestamp: 1715557231, protocol_version: Version22, storage_logs_chunks_processed: [true, false, true, false, true, true, true, true, true, false, true, true, true, false, false, true, false, true, false, true, true, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false] }
external-node-1  | 2024-05-13T18:10:05.710090Z  INFO zksync_external_node::init: Chosen node initialization strategy: SnapshotRecovery
external-node-1  | 2024-05-13T18:10:05.710094Z  WARN zksync_external_node::init: Proceeding with snapshot recovery. This is an experimental feature; use at your own risk
external-node-1  | 2024-05-13T18:10:05.740030Z  INFO zksync_snapshots_applier: Re-initialized snapshots applier after reset/failure in 2.520125ms
external-node-1  | 2024-05-13T18:10:05.741034Z  INFO zksync_snapshots_applier: Processing storage logs chunk 1
external-node-1  | 2024-05-13T18:10:05.741805Z  INFO zksync_snapshots_applier: Processing storage logs chunk 3
external-node-1  | 2024-05-13T18:10:05.741868Z  INFO zksync_snapshots_applier: Processing storage logs chunk 9
external-node-1  | 2024-05-13T18:10:05.741924Z  INFO zksync_snapshots_applier: Processing storage logs chunk 13
external-node-1  | 2024-05-13T18:10:05.741976Z  INFO zksync_snapshots_applier: Processing storage logs chunk 14
external-node-1  | 2024-05-13T18:10:05.742033Z  INFO zksync_snapshots_applier: Processing storage logs chunk 16
external-node-1  | 2024-05-13T18:10:05.742077Z  INFO zksync_snapshots_applier: Processing storage logs chunk 18
external-node-1  | 2024-05-13T18:10:05.742108Z  INFO zksync_snapshots_applier: Processing storage logs chunk 21
external-node-1  | 2024-05-13T18:10:05.742193Z  INFO zksync_snapshots_applier: Processing storage logs chunk 22
external-node-1  | 2024-05-13T18:10:05.742425Z  INFO zksync_snapshots_applier: Processing storage logs chunk 23
external-node-1  | 2024-05-13T18:11:07.336709Z  INFO zksync_snapshots_applier: Loaded 996982 storage logs from GCS for chunk 9 in 61.59648432s
external-node-1  | 2024-05-13T18:11:07.337231Z  INFO zksync_snapshots_applier: Loading 996982 storage logs into Postgres
external-node-1  | 2024-05-13T18:11:11.106609Z  INFO zksync_snapshots_applier: Loaded 998258 storage logs from GCS for chunk 22 in 65.366064822s
external-node-1  | 2024-05-13T18:11:11.107189Z  INFO zksync_snapshots_applier: Loading 998258 storage logs into Postgres
external-node-1  | 2024-05-13T18:11:14.320519Z  INFO zksync_snapshots_applier: Loaded 998477 storage logs from GCS for chunk 13 in 68.58024799s
external-node-1  | 2024-05-13T18:11:14.336580Z  INFO zksync_snapshots_applier: Loading 998477 storage logs into Postgres
external-node-1  | 2024-05-13T18:11:17.983291Z  INFO zksync_snapshots_applier: Loaded 999664 storage logs from GCS for chunk 21 in 72.242830284s
external-node-1  | 2024-05-13T18:11:17.988663Z  INFO zksync_snapshots_applier: Loading 999664 storage logs into Postgres
external-node-1  | 2024-05-13T18:11:20.573685Z  INFO zksync_snapshots_applier: Loaded 998916 storage logs from GCS for chunk 14 in 74.832843118s
external-node-1  | 2024-05-13T18:11:20.581494Z  INFO zksync_snapshots_applier: Loading 998916 storage logs into Postgres
external-node-1  | 2024-05-13T18:11:24.758088Z  INFO zksync_snapshots_applier: Loaded 1000153 storage logs from GCS for chunk 3 in 79.01740487s
external-node-1  | 2024-05-13T18:11:25.343647Z  INFO zksync_snapshots_applier: Loaded 997139 storage logs from GCS for chunk 16 in 79.602744995s
external-node-1  | 2024-05-13T18:11:25.366844Z  INFO zksync_snapshots_applier: Loading 1000153 storage logs into Postgres
external-node-1  | 2024-05-13T18:11:27.481143Z  INFO zksync_snapshots_applier: Loading 997139 storage logs into Postgres
grafana-1        | logger=cleanup t=2024-05-13T18:11:28.14106067Z level=info msg="Completed cleanup jobs" duration=278.648042ms
postgres-1       | 2024-05-13 18:11:29.735 UTC [10157] ERROR:  unexpected EOF on client connection with an open transaction
postgres-1       | 2024-05-13 18:11:29.735 UTC [10157] STATEMENT:  COPY storage_logs(
postgres-1       |                  hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1       |                  created_at, updated_at
postgres-1       |              )
postgres-1       |              FROM STDIN WITH (DELIMITER '|')
postgres-1       | 2024-05-13 18:11:29.735 UTC [10156] LOG:  incomplete message from client
postgres-1       | 2024-05-13 18:11:29.735 UTC [10156] STATEMENT:  COPY storage_logs(
postgres-1       |                  hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1       |                  created_at, updated_at
postgres-1       |              )
postgres-1       |              FROM STDIN WITH (DELIMITER '|')
postgres-1       | 2024-05-13 18:11:29.736 UTC [10156] ERROR:  unexpected EOF on client connection with an open transaction
postgres-1       | 2024-05-13 18:11:29.736 UTC [10156] STATEMENT:  COPY storage_logs(
postgres-1       |                  hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1       |                  created_at, updated_at
postgres-1       |              )
postgres-1       |              FROM STDIN WITH (DELIMITER '|')
postgres-1       | 2024-05-13 18:11:29.735 UTC [10119] LOG:  incomplete message from client
postgres-1       | 2024-05-13 18:11:29.735 UTC [10119] STATEMENT:  COPY storage_logs(
postgres-1       |                  hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1       |                  created_at, updated_at
postgres-1       |              )
postgres-1       |              FROM STDIN WITH (DELIMITER '|')
postgres-1       | 2024-05-13 18:11:29.736 UTC [10119] ERROR:  unexpected EOF on client connection with an open transaction
postgres-1       | 2024-05-13 18:11:29.736 UTC [10119] STATEMENT:  COPY storage_logs(
postgres-1       |                  hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1       |                  created_at, updated_at
postgres-1       |              )
postgres-1       |              FROM STDIN WITH (DELIMITER '|')
postgres-1       | 2024-05-13 18:11:29.737 UTC [10157] FATAL:  terminating connection because protocol synchronization was lost
postgres-1       | 2024-05-13 18:11:29.737 UTC [10156] FATAL:  terminating connection because protocol synchronization was lost
postgres-1       | 2024-05-13 18:11:29.740 UTC [10156] LOG:  could not send data to client: Broken pipe
postgres-1       | 2024-05-13 18:11:29.740 UTC [10157] LOG:  could not send data to client: Broken pipe
postgres-1       | 2024-05-13 18:11:29.753 UTC [10119] FATAL:  terminating connection because protocol synchronization was lost
postgres-1       | 2024-05-13 18:11:29.753 UTC [10119] LOG:  could not send data to client: Broken pipe
postgres-1       | 2024-05-13 18:11:29.768 UTC [9479] LOG:  unexpected EOF on client connection with an open transaction
external-node-1 exited with code 137
postgres-1       | 2024-05-13 18:11:33.349 UTC [10092] LOG:  unexpected EOF on client connection with an open transaction
postgres-1       | 2024-05-13 18:11:34.860 UTC [9481] LOG:  unexpected EOF on client connection with an open transaction
postgres-1       | 2024-05-13 18:11:36.116 UTC [10065] LOG:  unexpected EOF on client connection with an open transaction

Please advise. Thank you!

shaky commented 6 months ago

withdrawing this, I increased the RAM in docker, I'll observe and return in case of any other issues.