Bug: Connected peers on Gnosis chain drops to 0 every 5 minutes

ThomasDalla commented 1 year ago

Connected peers drops to zero every 5 minutes or so. Quickly increases back to 50ish after dropping.

Sample log around a drop:

lodestar_gnosis  | Nov-26 20:22:52.507[]                 info: Synced - slot: 6100166 - head: 6100166 0x4702…7e34 - finalized: 0xdae1…0490:381258 - peers: 50
lodestar_gnosis  | Nov-26 20:22:57.596[]                 info: Synced - slot: 6100167 - head: 6100167 0x0d70…2904 - finalized: 0xdae1…0490:381258 - peers: 50
lodestar_gnosis  | Nov-26 20:23:02.520[]                 info: Synced - slot: 6100168 - head: 6100168 0x1ee5…0ecd - finalized: 0xdae1…0490:381258 - peers: 51
lodestar_gnosis  | Nov-26 20:23:07.530[]                 info: Synced - slot: 6100169 - head: 6100169 0x38e3…f968 - finalized: 0xdae1…0490:381258 - peers: 51
lodestar_gnosis  | Nov-26 20:23:12.526[]                 info: Synced - slot: 6100170 - head: 6100170 0x5a71…54de - finalized: 0xdae1…0490:381258 - peers: 51
lodestar_gnosis  | Nov-26 20:23:17.780[]                 info: Synced - slot: 6100171 - head: 6100171 0x171f…75c7 - finalized: 0xdae1…0490:381258 - peers: 52
lodestar_gnosis  | Nov-26 20:23:22.795[]                 info: Synced - slot: 6100172 - head: 6100172 0xd519…d049 - finalized: 0xdae1…0490:381258 - peers: 52
lodestar_gnosis  | Nov-26 20:23:27.512[]                 info: Synced - slot: 6100173 - head: 6100173 0x43bf…0375 - finalized: 0xdae1…0490:381258 - peers: 53
lodestar_gnosis  | Nov-26 20:23:32.501[]                 warn: Low peer count peers=0
lodestar_gnosis  | Nov-26 20:23:32.501[]                 info: Searching peers - peers: 0 - slot: 6100174 - head: 6100174 0x09d4…0a63 - finalized: 0xdae1…0490:381258
lodestar_gnosis  | Nov-26 20:23:35.647[api]             error: Error on submitPoolSyncCommitteeSignatures [0] slot=6100175, validatorIndex=100567 PublishError.InsufficientPeers
lodestar_gnosis  | Error: PublishError.InsufficientPeers
lodestar_gnosis  |     at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
lodestar_gnosis  |     at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
lodestar_gnosis  |     at EventTarget.publishSyncCommitteeSignature (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:242:5)
lodestar_gnosis  |     at async Promise.all (index 0)
lodestar_gnosis  |     at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:146:13
lodestar_gnosis  |     at async Promise.all (index 0)
lodestar_gnosis  |     at Object.submitPoolSyncCommitteeSignatures (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:116:7)
lodestar_gnosis  |     at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
lodestar_gnosis  | Nov-26 20:23:35.648[rest]            error: Req req-3nw submitPoolSyncCommitteeSignatures error  PublishError.InsufficientPeers
lodestar_gnosis  | Error: PublishError.InsufficientPeers
lodestar_gnosis  |     at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
lodestar_gnosis  |     at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
lodestar_gnosis  |     at EventTarget.publishSyncCommitteeSignature (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:242:5)
lodestar_gnosis  |     at async Promise.all (index 0)
lodestar_gnosis  |     at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:146:13
lodestar_gnosis  |     at async Promise.all (index 0)
lodestar_gnosis  |     at Object.submitPoolSyncCommitteeSignatures (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:116:7)
lodestar_gnosis  |     at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
lodestar_gnosis  | Nov-26 20:23:37.501[]                 info: Synced - slot: 6100175 - head: 6100175 0xa2db…50eb - finalized: 0xdae1…0490:381258 - peers: 8
lodestar_gnosis  | Nov-26 20:23:42.501[]                 info: Synced - slot: 6100176 (skipped 1) - head: 6100175 0xa2db…50eb - finalized: 0x3083…d874:381259 - peers: 9
lodestar_gnosis  | Nov-26 20:23:47.500[]                 info: Synced - slot: 6100177 - head: 6100177 0xfaa1…771b - finalized: 0x3083…d874:381259 - peers: 8
lodestar_gnosis  | Nov-26 20:23:52.500[]                 info: Synced - slot: 6100178 - head: 6100178 0x3e83…4e89 - finalized: 0x3083…d874:381259 - peers: 11
lodestar_gnosis  | Nov-26 20:23:57.499[]                 info: Synced - slot: 6100179 (skipped 1) - head: 6100178 0x3e83…4e89 - finalized: 0x3083…d874:381259 - peers: 12
lodestar_gnosis  | Nov-26 20:24:02.529[]                 info: Synced - slot: 6100180 - head: 6100180 0x2d02…3a36 - finalized: 0x3083…d874:381259 - peers: 13
lodestar_gnosis  | Nov-26 20:24:07.500[]                 info: Synced - slot: 6100181 - head: 6100181 0x598c…7c5f - finalized: 0x3083…d874:381259 - peers: 14
lodestar_gnosis  | Nov-26 20:24:12.500[]                 info: Synced - slot: 6100182 - head: 6100182 0x25cb…321f - finalized: 0x3083…d874:381259 - peers: 14
lodestar_gnosis  | Nov-26 20:24:17.500[]                 info: Synced - slot: 6100183 - head: 6100183 0x2fdc…0677 - finalized: 0x3083…d874:381259 - peers: 14
lodestar_gnosis  | Nov-26 20:24:22.500[]                 info: Synced - slot: 6100184 - head: 6100184 0xf194…ddbf - finalized: 0x3083…d874:381259 - peers: 16

Lodestar running in docker compose, latest version.

    command: |
      beacon
      --network=gnosis
      --dataDir=/data
      --preset=gnosis
      --eth1=true
      --execution.urls=http://execution:8551
      --jwt-secret=/jwt.hex
      --logFile=/lodestar/beacon.log
      --logFileLevel=info
      --port=9002
      --rest=true
      --rest.address=0.0.0.0
      --rest.port=4000
      --rest.cors=*
      --discv5=true
      --targetPeers=50
      --metrics=true
      --metrics.port=5054
      --checkpointSyncUrl=https://checkpoint.gnosischain.com/
      --suggestedFeeRecipient=0xREDACTED

Desktop:

OS: Linux alderlake 5.15.0-53-generic #59-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
Version: v1.2.2
Branch: latest
Commit hash: f093b46

ThomasDalla commented 1 year ago

After a restart, it was stable for about 2 hours before having the same issue:

Beacon logs around the time the issue happened after the restart (9:36am):

Nov-27 09:34:52.530[]                 [32minfo[39m: Synced - slot: 6109670 (skipped 1) - head: 6109669 0xaa7c…cbef - finalized: 0xa085…3be0:381852 - peers: 50
Nov-27 09:34:57.514[]                 [32minfo[39m: Synced - slot: 6109671 - head: 6109671 0x2cac…141c - finalized: 0xa085…3be0:381852 - peers: 50
Nov-27 09:35:02.826[]                 [32minfo[39m: Synced - slot: 6109672 - head: 6109672 0xcdfc…0035 - finalized: 0xa085…3be0:381852 - peers: 51
Nov-27 09:35:07.660[]                 [32minfo[39m: Synced - slot: 6109673 - head: 6109673 0x9c8b…56e9 - finalized: 0xa085…3be0:381852 - peers: 52
Nov-27 09:35:12.529[]                 [32minfo[39m: Synced - slot: 6109674 - head: 6109674 0x11fe…16de - finalized: 0xa085…3be0:381852 - peers: 52
Nov-27 09:35:17.512[]                 [32minfo[39m: Synced - slot: 6109675 - head: 6109675 0x9767…deaf - finalized: 0xa085…3be0:381852 - peers: 50
Nov-27 09:35:22.524[]                 [32minfo[39m: Synced - slot: 6109676 - head: 6109676 0xd02e…21a6 - finalized: 0xa085…3be0:381852 - peers: 50
Nov-27 09:35:27.521[]                 [32minfo[39m: Synced - slot: 6109677 - head: 6109677 0xf065…fb49 - finalized: 0xa085…3be0:381852 - peers: 51
Nov-27 09:35:32.510[]                 [32minfo[39m: Synced - slot: 6109678 - head: 6109678 0x3c4d…ae60 - finalized: 0xa085…3be0:381852 - peers: 51
Nov-27 09:35:37.507[]                 [32minfo[39m: Synced - slot: 6109679 - head: 6109679 0x8f75…5327 - finalized: 0xa085…3be0:381852 - peers: 51
Nov-27 09:35:42.502[]                 [32minfo[39m: Synced - slot: 6109680 (skipped 1) - head: 6109679 0x8f75…5327 - finalized: 0x3ba1…cb76:381853 - peers: 50
Nov-27 09:35:47.543[]                 [32minfo[39m: Synced - slot: 6109681 - head: 6109681 0xb784…254d - finalized: 0x3ba1…cb76:381853 - peers: 50
Nov-27 09:35:52.782[]                 [32minfo[39m: Synced - slot: 6109682 - head: 6109682 0xc5e7…3bb1 - finalized: 0x3ba1…cb76:381853 - peers: 50
Nov-27 09:35:57.627[]                 [32minfo[39m: Synced - slot: 6109683 - head: 6109683 0x5816…8347 - finalized: 0x3ba1…cb76:381853 - peers: 51
Nov-27 09:36:02.503[]                 [32minfo[39m: Synced - slot: 6109684 (skipped 1) - head: 6109683 0x5816…8347 - finalized: 0x3ba1…cb76:381853 - peers: 51
Nov-27 09:36:07.511[]                 [32minfo[39m: Synced - slot: 6109685 - head: 6109685 0xb37f…bad4 - finalized: 0x3ba1…cb76:381853 - peers: 51
Nov-27 09:36:12.588[]                 [32minfo[39m: Synced - slot: 6109686 - head: 6109686 0x557a…fd59 - finalized: 0x3ba1…cb76:381853 - peers: 51
Nov-27 09:36:17.501[]                 [32minfo[39m: Synced - slot: 6109687 - head: 6109687 0x9635…f5da - finalized: 0x3ba1…cb76:381853 - peers: 50
Nov-27 09:36:23.120[]                 [32minfo[39m: Synced - slot: 6109688 - head: 6109688 0xa6f5…ca46 - finalized: 0x3ba1…cb76:381853 - peers: 51
Nov-27 09:36:27.695[]                 [32minfo[39m: Synced - slot: 6109689 - head: 6109689 0x5927…fc39 - finalized: 0x3ba1…cb76:381853 - peers: 53
Nov-27 09:36:32.545[]                 [32minfo[39m: Synced - slot: 6109690 - head: 6109690 0x994f…9c94 - finalized: 0x3ba1…cb76:381853 - peers: 55
Nov-27 09:36:37.559[]                 [32minfo[39m: Synced - slot: 6109691 - head: 6109691 0x25a3…8ddc - finalized: 0x3ba1…cb76:381853 - peers: 56
Nov-27 09:36:42.848[]                 [32minfo[39m: Synced - slot: 6109692 (skipped 1) - head: 6109691 0x25a3…8ddc - finalized: 0x3ba1…cb76:381853 - peers: 56
Nov-27 09:36:46.733[api]             [31merror[39m: Error on submitPoolSyncCommitteeSignatures [0] slot=6109693, validatorIndex=100626 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishSyncCommitteeSignature (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:242:5)
    at async Promise.all (index 0)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:146:13
    at async Promise.all (index 0)
    at Object.submitPoolSyncCommitteeSignatures (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:116:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:36:46.734[rest]            [31merror[39m: Req req-9sf submitPoolSyncCommitteeSignatures error  PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishSyncCommitteeSignature (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:242:5)
    at async Promise.all (index 0)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:146:13
    at async Promise.all (index 0)
    at Object.submitPoolSyncCommitteeSignatures (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:116:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:36:47.501[]                 [33mwarn[39m: Low peer count peers=1
Nov-27 09:36:47.501[]                 [32minfo[39m: Synced - slot: 6109693 (skipped 2) - head: 6109691 0x25a3…8ddc - finalized: 0x3ba1…cb76:381853 - peers: 1
Nov-27 09:36:52.499[]                 [32minfo[39m: Synced - slot: 6109694 (skipped 3) - head: 6109691 0x25a3…8ddc - finalized: 0x3ba1…cb76:381853 - peers: 4
Nov-27 09:36:57.499[]                 [32minfo[39m: Synced - slot: 6109695 - head: 6109695 0x68ab…62d9 - finalized: 0x3ba1…cb76:381853 - peers: 3
Nov-27 09:36:57.500[]                 [32minfo[39m: Synced - slot: 6109695 - head: 6109695 0x68ab…62d9 - finalized: 0x3ba1…cb76:381853 - peers: 3
Nov-27 09:37:02.500[]                 [32minfo[39m: Synced - slot: 6109696 - head: 6109696 0xcd73…e723 - finalized: 0x9e63…2187:381854 - peers: 3
Nov-27 09:37:05.557[api]             [31merror[39m: Error on submitPoolSyncCommitteeSignatures [0] slot=6109697, validatorIndex=100626 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishSyncCommitteeSignature (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:242:5)
    at async Promise.all (index 0)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:146:13
    at async Promise.all (index 0)
    at Object.submitPoolSyncCommitteeSignatures (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:116:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:05.558[rest]            [31merror[39m: Req req-9tc submitPoolSyncCommitteeSignatures error  PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishSyncCommitteeSignature (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:242:5)
    at async Promise.all (index 0)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:146:13
    at async Promise.all (index 0)
    at Object.submitPoolSyncCommitteeSignatures (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:116:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.396[api]             [31merror[39m: Error on submitPoolAttestations [0] slot=6109697, index=39 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 0)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.396[api]             [31merror[39m: Error on submitPoolAttestations [1] slot=6109697, index=48 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 1)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.407[api]             [31merror[39m: Error on submitPoolAttestations [3] slot=6109697, index=46 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 3)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.407[api]             [31merror[39m: Error on submitPoolAttestations [4] slot=6109697, index=43 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 4)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.408[api]             [31merror[39m: Error on submitPoolAttestations [5] slot=6109697, index=35 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 5)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.408[api]             [31merror[39m: Error on submitPoolAttestations [8] slot=6109697, index=2 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 8)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.408[api]             [31merror[39m: Error on submitPoolAttestations [9] slot=6109697, index=52 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 9)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.408[api]             [31merror[39m: Error on submitPoolAttestations [10] slot=6109697, index=4 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 10)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.408[api]             [31merror[39m: Error on submitPoolAttestations [11] slot=6109697, index=21 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 11)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.408[api]             [31merror[39m: Error on submitPoolAttestations [12] slot=6109697, index=43 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at EventTarget.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 12)
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:06.408[rest]            [31merror[39m: Req req-9td submitPoolAttestations error  Multiple errors on submitPoolAttestations
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
Error: Multiple errors on submitPoolAttestations
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:70:15)
    at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Nov-27 09:37:07.501[]                 [32minfo[39m: Synced - slot: 6109697 - head: 6109697 0x7eeb…7c11 - finalized: 0x9e63…2187:381854 - peers: 4
Nov-27 09:37:12.499[]                 [32minfo[39m: Synced - slot: 6109698 - head: 6109698 0x5eab…f7bc - finalized: 0x9e63…2187:381854 - peers: 6
Nov-27 09:37:12.501[]                 [32minfo[39m: Synced - slot: 6109698 - head: 6109698 0x5eab…f7bc - finalized: 0x9e63…2187:381854 - peers: 6
Nov-27 09:37:17.500[]                 [32minfo[39m: Synced - slot: 6109699 - head: 6109699 0x17db…dc68 - finalized: 0x9e63…2187:381854 - peers: 7
Nov-27 09:37:22.500[]                 [32minfo[39m: Synced - slot: 6109700 - head: 6109700 0x9b78…2e31 - finalized: 0x9e63…2187:381854 - peers: 8
Nov-27 09:37:27.499[]                 [32minfo[39m: Synced - slot: 6109701 - head: 6109701 0x7926…64c5 - finalized: 0x9e63…2187:381854 - peers: 7
Nov-27 09:37:32.500[]                 [32minfo[39m: Synced - slot: 6109702 - head: 6109702 0x87a0…3ac3 - finalized: 0x9e63…2187:381854 - peers: 7
Nov-27 09:37:37.500[]                 [32minfo[39m: Synced - slot: 6109703 - head: 6109703 0x1173…ecbf - finalized: 0x9e63…2187:381854 - peers: 7
Nov-27 09:37:42.500[]                 [32minfo[39m: Synced - slot: 6109704 - head: 6109704 0x1e0b…e195 - finalized: 0x9e63…2187:381854 - peers: 11
Nov-27 09:37:47.524[]                 [32minfo[39m: Synced - slot: 6109705 - head: 6109705 0x8b44…7776 - finalized: 0x9e63…2187:381854 - peers: 12
Nov-27 09:37:52.499[]                 [32minfo[39m: Synced - slot: 6109706 - head: 6109706 0x4d04…e21b - finalized: 0x9e63…2187:381854 - peers: 11
Nov-27 09:37:57.499[]                 [32minfo[39m: Synced - slot: 6109707 (skipped 1) - head: 6109706 0x4d04…e21b - finalized: 0x9e63…2187:381854 - peers: 12
Nov-27 09:37:57.501[]                 [32minfo[39m: Synced - slot: 6109707 (skipped 1) - head: 6109706 0x4d04…e21b - finalized: 0x9e63…2187:381854 - peers: 12
Nov-27 09:38:02.512[]                 [32minfo[39m: Synced - slot: 6109708 - head: 6109708 0x96d7…979a - finalized: 0x9e63…2187:381854 - peers: 12
Nov-27 09:38:07.500[]                 [32minfo[39m: Synced - slot: 6109709 (skipped 1) - head: 6109708 0x96d7…979a - finalized: 0x9e63…2187:381854 - peers: 12
Nov-27 09:38:12.506[]                 [32minfo[39m: Synced - slot: 6109710 - head: 6109710 0xd7b6…8aa4 - finalized: 0x9e63…2187:381854 - peers: 12
Nov-27 09:38:17.566[]                 [32minfo[39m: Synced - slot: 6109711 (skipped 1) - head: 6109710 0xd7b6…8aa4 - finalized: 0x9e63…2187:381854 - peers: 13
Nov-27 09:38:22.520[]                 [32minfo[39m: Synced - slot: 6109712 - head: 6109712 0x2c53…0f31 - finalized: 0x8f75…5327:381855 - peers: 15
Nov-27 09:38:27.534[]                 [32minfo[39m: Synced - slot: 6109713 - head: 6109713 0xd752…ee35 - finalized: 0x8f75…5327:381855 - peers: 14
Nov-27 09:38:32.571[]                 [32minfo[39m: Synced - slot: 6109714 - head: 6109714 0xc9c1…591d - finalized: 0x8f75…5327:381855 - peers: 16
Nov-27 09:38:37.500[]                 [32minfo[39m: Synced - slot: 6109715 - head: 6109715 0x213b…cd5d - finalized: 0x8f75…5327:381855 - peers: 16
Nov-27 09:38:42.517[]                 [32minfo[39m: Synced - slot: 6109716 - head: 6109716 0x2f21…ace7 - finalized: 0x8f75…5327:381855 - peers: 17
Nov-27 09:38:47.616[]                 [32minfo[39m: Synced - slot: 6109717 - head: 6109717 0x043f…2421 - finalized: 0x8f75…5327:381855 - peers: 17
Nov-27 09:38:52.501[]                 [32minfo[39m: Synced - slot: 6109718 - head: 6109718 0x35e0…91ab - finalized: 0x8f75…5327:381855 - peers: 17
Nov-27 09:38:57.513[]                 [32minfo[39m: Synced - slot: 6109719 - head: 6109719 0x603c…9b46 - finalized: 0x8f75…5327:381855 - peers: 15
Nov-27 09:39:02.607[]                 [32minfo[39m: Synced - slot: 6109720 - head: 6109720 0x5aef…0b49 - finalized: 0x8f75…5327:381855 - peers: 15
Nov-27 09:39:07.506[]                 [32minfo[39m: Synced - slot: 6109721 - head: 6109721 0xe1aa…0a35 - finalized: 0x8f75…5327:381855 - peers: 16
Nov-27 09:39:12.501[]                 [32minfo[39m: Synced - slot: 6109722 - head: 6109722 0xf727…0b88 - finalized: 0x8f75…5327:381855 - peers: 16
Nov-27 09:39:17.515[]                 [32minfo[39m: Synced - slot: 6109723 - head: 6109723 0x7f4f…a1c6 - finalized: 0x8f75…5327:381855 - peers: 16
Nov-27 09:39:22.501[]                 [32minfo[39m: Synced - slot: 6109724 (skipped 1) - head: 6109723 0x7f4f…a1c6 - finalized: 0x8f75…5327:381855 - peers: 20
Nov-27 09:39:27.564[]                 [32minfo[39m: Synced - slot: 6109725 - head: 6109725 0x3273…8376 - finalized: 0x8f75…5327:381855 - peers: 19

Logs of the validator client around the same time:

Nov-27 09:36:35.637[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109691, count=1
Nov-27 09:36:36.614[]                 [32minfo[39m: Published attestations slot=6109691, count=16
Nov-27 09:36:41.794[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109692, count=1
Nov-27 09:36:41.940[]                 [32minfo[39m: Published attestations slot=6109692, count=11
Nov-27 09:36:43.594[]                 [32minfo[39m: Published aggregateAndProofs slot=6109692, index=17, count=1
Nov-27 09:36:43.594[]                 [32minfo[39m: Published aggregateAndProofs slot=6109692, index=34, count=1
Nov-27 09:36:45.004[]                 [32minfo[39m: Node is syncing - Service Unavailable: Node is syncing - waiting for peers - Error on getProposerDuties
Nov-27 09:36:46.667[]                [31merror[39m: Error on runEvery fn  Service Unavailable: Node is syncing - waiting for peers - Error producing attestation
Error: Service Unavailable: Node is syncing - waiting for peers - Error producing attestation
    at HttpClient.requestWithBody (file:///usr/app/packages/api/src/utils/client/httpClient.ts:258:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at HttpClient.json (file:///usr/app/packages/api/src/utils/client/httpClient.ts:128:12)
    at Object.request [as produceAttestationData] (file:///usr/app/packages/api/src/utils/client/client.ts:75:21)
    at AttestationService.produceAttestation (file:///usr/app/packages/validator/src/services/attestation.ts:96:28)
    at AttestationService.runAttestationTasks (file:///usr/app/packages/validator/src/services/attestation.ts:62:36)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Nov-27 09:36:46.736[]                [31merror[39m: Error publishing SyncCommitteeMessage slot=6109693 Internal Server Error: PublishError.InsufficientPeers
Error: Internal Server Error: PublishError.InsufficientPeers
    at HttpClient.requestWithBody (file:///usr/app/packages/api/src/utils/client/httpClient.ts:258:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at HttpClient.request (file:///usr/app/packages/api/src/utils/client/httpClient.ts:132:12)
    at Object.request [as submitPoolSyncCommitteeSignatures] (file:///usr/app/packages/api/src/utils/client/client.ts:82:9)
    at SyncCommitteeService.produceAndPublishSyncCommittees (file:///usr/app/packages/validator/src/services/syncCommittee.ts:148:9)
    at SyncCommitteeService.runSyncCommitteeTasks (file:///usr/app/packages/validator/src/services/syncCommittee.ts:67:31)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Nov-27 09:36:51.774[]                 [32minfo[39m: Published attestations slot=6109694, count=8
Nov-27 09:36:51.774[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109694, count=1
Nov-27 09:36:53.459[]                 [32minfo[39m: Published aggregateAndProofs slot=6109694, index=17, count=1
Nov-27 09:36:53.459[]                 [32minfo[39m: Published aggregateAndProofs slot=6109694, index=0, count=1
Nov-27 09:36:56.560[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109695, count=1
Nov-27 09:36:56.704[]                 [32minfo[39m: Published attestations slot=6109695, count=15
Nov-27 09:36:58.557[]                 [32minfo[39m: Published aggregateAndProofs slot=6109695, index=23, count=1
Nov-27 09:36:58.557[]                 [32minfo[39m: Published aggregateAndProofs slot=6109695, index=47, count=1
Nov-27 09:36:58.558[]                 [32minfo[39m: Published aggregateAndProofs slot=6109695, index=2, count=1
Nov-27 09:37:00.597[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109696, count=1
Nov-27 09:37:01.464[]                 [32minfo[39m: Published attestations slot=6109696, count=18
Nov-27 09:37:03.379[]                 [32minfo[39m: Published aggregateAndProofs slot=6109696, index=51, count=1
Nov-27 09:37:05.558[]                [31merror[39m: Error publishing SyncCommitteeMessage slot=6109697 Internal Server Error: PublishError.InsufficientPeers
Error: Internal Server Error: PublishError.InsufficientPeers
    at HttpClient.requestWithBody (file:///usr/app/packages/api/src/utils/client/httpClient.ts:258:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at HttpClient.request (file:///usr/app/packages/api/src/utils/client/httpClient.ts:132:12)
    at Object.request [as submitPoolSyncCommitteeSignatures] (file:///usr/app/packages/api/src/utils/client/client.ts:82:9)
    at SyncCommitteeService.produceAndPublishSyncCommittees (file:///usr/app/packages/validator/src/services/syncCommittee.ts:148:9)
    at SyncCommitteeService.runSyncCommitteeTasks (file:///usr/app/packages/validator/src/services/syncCommittee.ts:67:31)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Nov-27 09:37:06.409[]                [31merror[39m: Error publishing attestations slot=6109697 Internal Server Error: Multiple errors on submitPoolAttestations
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
Error: Internal Server Error: Multiple errors on submitPoolAttestations
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
PublishError.InsufficientPeers
    at HttpClient.requestWithBody (file:///usr/app/packages/api/src/utils/client/httpClient.ts:258:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at HttpClient.request (file:///usr/app/packages/api/src/utils/client/httpClient.ts:132:12)
    at Object.request [as submitPoolAttestations] (file:///usr/app/packages/api/src/utils/client/client.ts:82:9)
    at AttestationService.signAndPublishAttestations (file:///usr/app/packages/validator/src/services/attestation.ts:146:7)
    at AttestationService.runAttestationTasks (file:///usr/app/packages/validator/src/services/attestation.ts:65:5)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Nov-27 09:37:08.386[]                 [32minfo[39m: Published aggregateAndProofs slot=6109697, index=41, count=1
Nov-27 09:37:10.730[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109698, count=1
Nov-27 09:37:11.576[]                 [32minfo[39m: Published attestations slot=6109698, count=19
Nov-27 09:37:13.367[]                 [32minfo[39m: Published aggregateAndProofs slot=6109698, index=50, count=1
Nov-27 09:37:13.398[]                 [32minfo[39m: Published aggregateAndProofs slot=6109698, index=23, count=1
Nov-27 09:37:16.065[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109699, count=1
Nov-27 09:37:16.718[]                 [32minfo[39m: Published attestations slot=6109699, count=16
Nov-27 09:37:18.360[]                 [32minfo[39m: Published aggregateAndProofs slot=6109699, index=21, count=1
Nov-27 09:37:18.389[]                 [32minfo[39m: Published aggregateAndProofs slot=6109699, index=41, count=1
Nov-27 09:37:20.652[]                 [32minfo[39m: Published SyncCommitteeMessage slot=6109700, count=1
Nov-27 09:37:21.408[]                 [32minfo[39m: Published attestations slot=6109700, count=15
Nov-27 09:37:23.378[]                 [32minfo[39m: Published SyncCommitteeContribution slot=6109700, index=2, count=1

ThomasDalla commented 1 year ago

Another one, logs in debug mode this time.

Starts around Nov-27 14:46:16:


Nov-27 14:46:16.241[network]       [36mverbose[39m: Req  error method=ping, encoding=ssz_snappy, client=Unknown, peer=16...rhDhGC, requestId=722 The operation was aborted
Error: The operation was aborted
    at nextAbortHandler (file:///usr/app/node_modules/abortable-iterator/src/index.ts:37:20)
    at AbortSignal.abortHandler (file:///usr/app/node_modules/abortable-iterator/src/index.ts:21:37)
    at AbortSignal.[nodejs.internal.kHybridDispatch] (node:internal/event_target:736:20)
    at AbortSignal.dispatchEvent (node:internal/event_target:678:26)
    at abortSignal (node:internal/abort_controller:292:10)
    at AbortController.abort (node:internal/abort_controller:323:5)
    at AbortSignal.onAbort (/usr/app/node_modules/any-signal/index.js:13:16)
    at AbortSignal.[nodejs.internal.kHybridDispatch] (node:internal/event_target:736:20)
    at AbortSignal.dispatchEvent (node:internal/event_target:678:26)
    at abortSignal (node:internal/abort_controller:292:10)
Nov-27 14:46:16.244[network]       [36mverbose[39m: Req  error method=status, encoding=ssz_snappy, client=Unknown, peer=16...rhDhGC, requestId=723 code=REQUEST_ERROR_DIAL_ERROR, error=The operation was aborted
Error: The operation was aborted
    at nextAbortHandler (file:///usr/app/node_modules/abortable-iterator/src/index.ts:37:20)
    at AbortSignal.abortHandler (file:///usr/app/node_modules/abortable-iterator/src/index.ts:21:37)
    at AbortSignal.[nodejs.internal.kHybridDispatch] (node:internal/event_target:736:20)
    at AbortSignal.dispatchEvent (node:internal/event_target:678:26)
    at abortSignal (node:internal/abort_controller:292:10)
    at AbortController.abort (node:internal/abort_controller:323:5)
    at AbortSignal.onAbort (/usr/app/node_modules/any-signal/index.js:13:16)
    at AbortSignal.[nodejs.internal.kHybridDispatch] (node:internal/event_target:736:20)
    at AbortSignal.dispatchEvent (node:internal/event_target:678:26)
    at abortSignal (node:internal/abort_controller:292:10)
Error: REQUEST_ERROR_DIAL_ERROR
    at file:///usr/app/packages/beacon-node/src/network/reqresp/request/index.ts:102:15
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at sendRequest (file:///usr/app/packages/beacon-node/src/network/reqresp/request/index.ts:88:20)
    at ReqResp.sendRequest (file:///usr/app/packages/beacon-node/src/network/reqresp/reqResp.ts:198:22)
    at ReqResp.status (file:///usr/app/packages/beacon-node/src/network/reqresp/reqResp.ts:96:12)
    at PeerManager.requestStatus (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:360:27)
Nov-27 14:46:16.245[network]       [36mverbose[39m: peer disconnected peer=16...rhDhGC, direction=outbound, status=CLOSED
``

twoeths commented 1 year ago

@dapplion @philknows should we add some nodes to track gnosis? I don't know how to proceed with this issue without a running gnosis node

philknows commented 1 year ago

@dapplion @philknows should we add some nodes to track gnosis? I don't know how to proceed with this issue without a running gnosis node

It's currently not a priority in terms of roadmap or resource usage... however, I'm open to the idea of dedicating resources to it. Maintaining this for one chain is already a lot of work in itself.

ThomasDalla commented 1 year ago

I tried again in case it would be better now, but still facing the same issue.

lodestar_gnosis  | Dec-30 01:31:57.501[]                 info: Synced - slot: 6673395 - head: 6673395 0xba60…c0a1 - execution: valid(0x9250…c152) - finalized: 0xce20…d4de:417085 - peers: 40
lodestar_gnosis  | Dec-30 01:32:02.501[]                 info: Synced - slot: 6673396 - head: 6673396 0xf86d…7722 - execution: valid(0x4092…e9c9) - finalized: 0xce20…d4de:417085 - peers: 39
lodestar_gnosis  | Dec-30 01:32:07.500[]                 info: Synced - slot: 6673397 - head: 6673397 0x191e…2943 - execution: valid(0x0c6f…1a99) - finalized: 0xce20…d4de:417085 - peers: 39
lodestar_gnosis  | Dec-30 01:32:12.503[]                 info: Synced - slot: 6673398 - head: 6673398 0x7cf4…8f05 - execution: valid(0x9a4c…d1a6) - finalized: 0xce20…d4de:417085 - peers: 49
lodestar_gnosis  | Dec-30 01:32:17.501[]                 warn: Low peer count peers=0
lodestar_gnosis  | Dec-30 01:32:17.501[]                 info: Searching peers - peers: 0 - slot: 6673399 (skipped 1) - head: 6673398 0x7cf4…8f05 - execution: valid(0x9a4c…d1a6) - finalized: 0xce20…d4de:417085
lodestar_gnosis  | Dec-30 01:32:22.501[]                 info: Synced - slot: 6673400 (skipped 2) - head: 6673398 0x7cf4…8f05 - execution: valid(0x9a4c…d1a6) - finalized: 0xce20…d4de:417085 - peers: 3
lodestar_gnosis  | Dec-30 01:32:27.500[]                 info: Synced - slot: 6673401 - head: 6673401 0xc2d8…4881 - execution: valid(0x2f8d…965c) - finalized: 0xce20…d4de:417085 - peers: 3
lodestar_gnosis  | Dec-30 01:32:32.501[]                 info: Synced - slot: 6673402 - head: 6673402 0xc9a9…9c3b - execution: valid(0x2b19…03c3) - finalized: 0xce20…d4de:417085 - peers: 4
lodestar_gnosis  | Dec-30 01:32:37.500[]                 info: Synced - slot: 6673403 - head: 6673403 0x7d62…8ee9 - execution: valid(0x71e6…ea61) - finalized: 0xce20…d4de:417085 - peers: 5
lodestar_gnosis  | Dec-30 01:32:42.499[]                 info: Synced - slot: 6673404 - head: 6673404 0xe710…0119 - execution: valid(0x1f72…74c3) - finalized: 0xce20…d4de:417085 - peers: 6

Because of that peers drop, lodestar is performing very bad for Gnosis chain and not suitable for running validators, despite being marked as "supported".

wemeetagain commented 1 year ago

Can you confirm if this still happens with our latest v1.4.2 release?

ThomasDalla commented 1 year ago

Just tried and it seems to still be happening.

logs | grep peers

lodestar_gnosis  | Feb-10 19:36:02.508[]                 info: Synced - slot: 7412164 - head: 7412164 0x5c9b…1015 - execution: valid(0x5ad2…9e26) - finalized: 0x29d2…94ab:463258 - peers: 51
lodestar_gnosis  | Feb-10 19:36:07.504[]                 info: Synced - slot: 7412165 - head: 7412165 0x94db…c7df - execution: valid(0x3e67…71d0) - finalized: 0x29d2…94ab:463258 - peers: 51
lodestar_gnosis  | Feb-10 19:36:12.505[]                 info: Synced - slot: 7412166 - head: 7412166 0x29c9…3959 - execution: valid(0xb133…d8b2) - finalized: 0x29d2…94ab:463258 - peers: 51
lodestar_gnosis  | Feb-10 19:36:17.499[]                 info: Synced - slot: 7412167 - head: 7412167 0x7e97…541d - execution: valid(0x1b00…ade5) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:17.507[]                 info: Synced - slot: 7412167 - head: 7412167 0x7e97…541d - execution: valid(0x1b00…ade5) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:22.504[]                 info: Synced - slot: 7412168 - head: 7412168 0x5148…9389 - execution: valid(0xeafe…2ce8) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:27.507[]                 info: Synced - slot: 7412169 - head: 7412169 0xd32b…078c - execution: valid(0xbf6f…6700) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:32.499[]                 info: Synced - slot: 7412170 - head: 7412170 0x12cb…a5b2 - execution: valid(0x643f…3a80) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:32.501[]                 info: Synced - slot: 7412170 - head: 7412170 0x12cb…a5b2 - execution: valid(0x643f…3a80) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:37.513[]                 info: Synced - slot: 7412171 - head: 7412171 0x1baa…5a92 - execution: valid(0xb485…4741) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:42.500[]                 info: Synced - slot: 7412172 - head: 7412172 0x54c1…2700 - execution: valid(0x49dd…1839) - finalized: 0x29d2…94ab:463258 - peers: 50
lodestar_gnosis  | Feb-10 19:36:47.680[]                 info: Synced - slot: 7412173 - head: 7412173 0x8235…7cd5 - execution: valid(0x641e…c6c1) - finalized: 0x29d2…94ab:463258 - peers: 55
lodestar_gnosis  | Feb-10 19:36:52.930[]                 info: Synced - slot: 7412174 - head: 7412174 0xcd1f…a1e8 - execution: valid(0x8eea…9660) - finalized: 0x29d2…94ab:463258 - peers: 54
lodestar_gnosis  | Feb-10 19:36:57.511[]                 warn: Low peer count peers=0
lodestar_gnosis  | Feb-10 19:36:57.511[]                 info: Searching peers - peers: 0 - slot: 7412175 - head: 7412175 0xc6f0…cd89 - execution: valid(0x70c3…478a) - finalized: 0x29d2…94ab:463258
lodestar_gnosis  | Feb-10 19:37:02.502[]                 info: Synced - slot: 7412176 (skipped 1) - head: 7412175 0xc6f0…cd89 - execution: valid(0x70c3…478a) - finalized: 0x9cd1…d630:463259 - peers: 2
lodestar_gnosis  | Feb-10 19:37:07.500[]                 info: Synced - slot: 7412177 - head: 7412177 0x1739…b87c - execution: valid(0x4fe6…dfc1) - finalized: 0x9cd1…d630:463259 - peers: 4
lodestar_gnosis  | Feb-10 19:37:12.501[]                 info: Synced - slot: 7412178 - head: 7412178 0x8b8f…a209 - execution: valid(0x38d2…76be) - finalized: 0x9cd1…d630:463259 - peers: 4
lodestar_gnosis  | Feb-10 19:37:17.532[]                 info: Synced - slot: 7412179 - head: 7412179 0x0f06…e944 - execution: valid(0xac53…9901) - finalized: 0x9cd1…d630:463259 - peers: 8
lodestar_gnosis  | Feb-10 19:37:22.532[]                 info: Synced - slot: 7412180 - head: 7412180 0x4223…ca79 - execution: valid(0x2d00…23a0) - finalized: 0x9cd1…d630:463259 - peers: 9
lodestar_gnosis  | Feb-10 19:37:27.500[]                 info: Synced - slot: 7412181 - head: 7412181 0xe485…73e6 - execution: valid(0x1a11…1461) - finalized: 0x9cd1…d630:463259 - peers: 8
lodestar_gnosis  | Feb-10 19:37:32.896[]                 info: Synced - slot: 7412182 - head: 7412182 0xda70…dc02 - execution: valid(0x5431…216f) - finalized: 0x9cd1…d630:463259 - peers: 9
lodestar_gnosis  | Feb-10 19:37:37.501[]                 info: Synced - slot: 7412183 - head: 7412183 0x1ece…3629 - execution: valid(0x6d7d…2259) - finalized: 0x9cd1…d630:463259 - peers: 8
lodestar_gnosis  | Feb-10 19:37:42.499[]                 info: Synced - slot: 7412184 - head: 7412184 0xb98b…53d6 - execution: valid(0x19d7…29e6) - finalized: 0x9cd1…d630:463259 - peers: 8
lodestar_gnosis  | Feb-10 19:37:42.501[]                 info: Synced - slot: 7412184 - head: 7412184 0xb98b…53d6 - execution: valid(0x19d7…29e6) - finalized: 0x9cd1…d630:463259 - peers: 8
lodestar_gnosis  | Feb-10 19:37:47.545[]                 info: Synced - slot: 7412185 - head: 7412185 0x7b59…fca9 - execution: valid(0x02a2…a910) - finalized: 0x9cd1…d630:463259 - peers: 19
lodestar_gnosis  | Feb-10 19:37:52.519[]                 info: Synced - slot: 7412186 - head: 7412186 0xbf67…29b5 - execution: valid(0xa95d…79cf) - finalized: 0x9cd1…d630:463259 - peers: 20
lodestar_gnosis  | Feb-10 19:37:57.515[]                 info: Synced - slot: 7412187 - head: 7412187 0xd129…6420 - execution: valid(0x1fdc…91f6) - finalized: 0x9cd1…d630:463259 - peers: 20
lodestar_gnosis  | Feb-10 19:38:02.557[]                 info: Synced - slot: 7412188 - head: 7412188 0x0c00…f332 - execution: valid(0x0c89…6ca9) - finalized: 0x9cd1…d630:463259 - peers: 20
lodestar_gnosis  | Feb-10 19:38:07.508[]                 info: Synced - slot: 7412189 - head: 7412189 0x61db…8574 - execution: valid(0x6e89…2ca9) - finalized: 0x9cd1…d630:463259 - peers: 20

I also see a lot of publish errors:

lodestar_gnosis  | Feb-10 18:41:06.150[api]             error: Error on submitPoolAttestations [12] slot=7411505, index=2 PublishError.Duplicate
lodestar_gnosis  | Error: PublishError.Duplicate
lodestar_gnosis  |     at Eth2Gossipsub.publish (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:2019:13)
lodestar_gnosis  |     at Eth2Gossipsub.publishObject (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
lodestar_gnosis  |     at Eth2Gossipsub.publishBeaconAttestation (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:221:12)
lodestar_gnosis  |     at file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:57:31
lodestar_gnosis  |     at async Promise.all (index 12)
lodestar_gnosis  |     at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:51:7)
lodestar_gnosis  |     at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:45:23)

I think that's because my validator client (teku) is setup with 2 beacon nodes (teku and lodestar). I was expecting it to work as a failover, not as a "duplicate"! Will try running the vc without the failover with lodestar see if that improves the peers connection on lodestar.

ThomasDalla commented 1 year ago

OK, even without any validator client connected to the beacon node, the peers keep dropping.

Verbose logs around one of the drops:

Feb-10 20:00:30.000[chain]         [36mverbose[39m: Clock slot slot=7412458
Feb-10 20:00:30.025[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...WbA5vc, requestId=1619, body=0
Feb-10 20:00:30.026[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...WbA5vc, requestId=1619
Feb-10 20:00:30.230[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...EaSdma, requestId=1620, body=11357
Feb-10 20:00:30.232[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...EaSdma, requestId=1620
Feb-10 20:00:30.560[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23680576, toBlock=23681575
Feb-10 20:00:30.583[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23681576, toBlock=23682575
Feb-10 20:00:30.987[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...JEWMCj, requestId=1622, body=0
Feb-10 20:00:30.988[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...JEWMCj, requestId=1622
Feb-10 20:00:31.076[network]         [34mdebug[39m: Resp received request method=status, client=unknown, peer=16...t6XGxA, requestId=1621, body=undefined
Feb-10 20:00:31.077[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...t6XGxA, requestId=1623, body=3883
Feb-10 20:00:31.078[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...t6XGxA, requestId=1623
Feb-10 20:00:31.078[network]       [36mverbose[39m: Resp done method=status, client=unknown, peer=16...t6XGxA, requestId=1621
Feb-10 20:00:31.134[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...LYYtqn, requestId=1624, body=2614
Feb-10 20:00:31.135[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...LYYtqn, requestId=1624
Feb-10 20:00:31.341[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23681576, toBlock=23682575
Feb-10 20:00:31.377[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23682576, toBlock=23683575
Feb-10 20:00:31.847[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...EZLaj4, requestId=1625, body=0
Feb-10 20:00:31.848[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...EZLaj4, requestId=1625
Feb-10 20:00:32.024[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23682576, toBlock=23683575
Feb-10 20:00:32.042[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23683576, toBlock=23684575
Feb-10 20:00:32.278[network]         [34mdebug[39m: Error dialing discovered peer peer=16...vVcWAQ All promises were rejected
AggregateError: All promises were rejected
Feb-10 20:00:32.500[]                 [32minfo[39m: Synced - slot: 7412458 (skipped 1) - head: 7412457 0x8948…4e70 - execution: valid(0x7676…da9f) - finalized: 0x4d6f…a462:463276 - peers: 28
Feb-10 20:00:32.523[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...7hZxNm, requestId=1626, body=3484
Feb-10 20:00:32.523[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...7hZxNm, requestId=1626
Feb-10 20:00:32.677[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23683576, toBlock=23684575
Feb-10 20:00:32.695[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23684576, toBlock=23685575
Feb-10 20:00:33.333[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23684576, toBlock=23685575
Feb-10 20:00:33.347[chain]         [36mverbose[39m: Running prepareForNextSlot nextEpoch=463279, prepareSlot=7412459, headSlot=7412457, headRoot=0x8948becc1d3b36f4ac517d52daaaa103210c8807c4c069d81e34682a33fe4e70, isEpochTransition=false
Feb-10 20:00:33.370[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23685576, toBlock=23686575
Feb-10 20:00:34.156[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23685576, toBlock=23686575
Feb-10 20:00:34.191[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23686576, toBlock=23687575
Feb-10 20:00:34.557[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...FfGQ5M, requestId=1628, body=0
Feb-10 20:00:34.558[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...FfGQ5M, requestId=1628
Feb-10 20:00:34.815[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...LUDWEv, requestId=1627, body=43770
Feb-10 20:00:34.817[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...LUDWEv, requestId=1627
Feb-10 20:00:34.862[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...NQNTGi, requestId=1629, body=163
Feb-10 20:00:34.862[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...NQNTGi, requestId=1629
Feb-10 20:00:34.892[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23686576, toBlock=23687575
Feb-10 20:00:34.940[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23687576, toBlock=23688575
Feb-10 20:00:34.966[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...g1RX9L, requestId=1630, body=0
Feb-10 20:00:34.966[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...g1RX9L, requestId=1630
Feb-10 20:00:35.000[chain]         [36mverbose[39m: Clock slot slot=7412459
Feb-10 20:00:35.106[rest]            [34mdebug[39m: Req req-5 100.77.38.70 getPeerCount
Feb-10 20:00:35.110[rest]            [34mdebug[39m: Res req-5 getPeerCount - 200
Feb-10 20:00:35.456[network]       [36mverbose[39m: Received gossip block slot=7412459, root=0x7b7f…34d5, curentSlot=7412459, peerId=16Uiu2HAmCCkevF1AWXV1kYt4Sv8QoSu51xMXMAWZuaCAymMW8xf1, delaySec=0.4440000057220459, recvToVal=0.012000083923339844
Feb-10 20:00:35.533[chain]         [36mverbose[39m: Transitioned gossip block slot=7412459, recvToTransition=0.08899998664855957
Feb-10 20:00:35.566[chain]         [36mverbose[39m: Verified block signatures slot=7412459, recvToSigVer=0.12199997901916504
Feb-10 20:00:35.583[chain]         [36mverbose[39m: Verified execution payload slot=7412459, recvToVerifiedExecPayload=0.13899993896484375
Feb-10 20:00:35.635[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23687576, toBlock=23688575
Feb-10 20:00:35.648[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...cEh4hM, requestId=1631, body=68
Feb-10 20:00:35.648[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...cEh4hM, requestId=1631
Feb-10 20:00:35.653[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23688576, toBlock=23689575
Feb-10 20:00:35.757[chain]           [34mdebug[39m: Persisted block to hot DB slot=7412459, root=0x7b7f6d83abc57d259f0f13b05be99da19ec5740e5197bdbf20aa3a137f8d34d5
Feb-10 20:00:35.759[chain]         [36mverbose[39m: Added block to forkchoice slot=7412459, root=0x7b7f6d83abc57d259f0f13b05be99da19ec5740e5197bdbf20aa3a137f8d34d5
Feb-10 20:00:35.782[chain]         [36mverbose[39m: New chain head slot=7412459, root=0x7b7f6d83abc57d259f0f13b05be99da19ec5740e5197bdbf20aa3a137f8d34d5, delaySec=0.7820000648498535
Feb-10 20:00:35.783[chain]         [36mverbose[39m: Imported block slot=7412459, recvToImportedBlock=0.33899998664855957
Feb-10 20:00:35.783[chain]         [36mverbose[39m: Block processed slot=7412459, root=0x7b7f6d83abc57d259f0f13b05be99da19ec5740e5197bdbf20aa3a137f8d34d5, delaySec=0.7829999923706055
Feb-10 20:00:36.417[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23688576, toBlock=23689575
Feb-10 20:00:36.435[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23689576, toBlock=23690575
Feb-10 20:00:37.091[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23689576, toBlock=23690575
Feb-10 20:00:37.112[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23690576, toBlock=23691575
Feb-10 20:00:37.500[]                 [32minfo[39m: Synced - slot: 7412459 - head: 7412459 0x7b7f…34d5 - execution: valid(0x3983…dad1) - finalized: 0x4d6f…a462:463276 - peers: 28
Feb-10 20:00:37.889[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...pdwzti, requestId=1632, body=0
Feb-10 20:00:37.890[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...pdwzti, requestId=1632
Feb-10 20:00:37.937[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23690576, toBlock=23691575
Feb-10 20:00:37.972[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23691576, toBlock=23692575
Feb-10 20:00:38.344[chain]         [36mverbose[39m: Running prepareForNextSlot nextEpoch=463279, prepareSlot=7412460, headSlot=7412459, headRoot=0x7b7f6d83abc57d259f0f13b05be99da19ec5740e5197bdbf20aa3a137f8d34d5, isEpochTransition=false
Feb-10 20:00:38.382[network]         [34mdebug[39m: Dialing discovered peer peer=16...PChr7U
Feb-10 20:00:38.399[network]         [34mdebug[39m: Dialing discovered peer peer=16...wRz8C6
Feb-10 20:00:38.404[network]         [34mdebug[39m: Dialing discovered peer peer=16...7S4Fkg
Feb-10 20:00:38.419[network]         [34mdebug[39m: Dialing discovered peer peer=16...qavN3T
Feb-10 20:00:38.420[network]         [34mdebug[39m: Dialing discovered peer peer=16...kjGNhe
Feb-10 20:00:38.432[network]         [34mdebug[39m: Dialing discovered peer peer=16...ccUEXM
Feb-10 20:00:38.439[network]         [34mdebug[39m: Dialing discovered peer peer=16...qavN3T
Feb-10 20:00:38.500[network]         [34mdebug[39m: Dialing discovered peer peer=16...33BGkB
Feb-10 20:00:38.566[network]         [34mdebug[39m: Dialing discovered peer peer=16...uZp5Qz
Feb-10 20:00:38.653[network]         [34mdebug[39m: Dialing discovered peer peer=16...9Sj5HX
Feb-10 20:00:38.699[network]         [34mdebug[39m: Dialed discovered peer peer=16...PChr7U
Feb-10 20:00:38.700[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...PChr7U, requestId=1633, body=undefined
Feb-10 20:00:38.701[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...PChr7U, requestId=1633
Feb-10 20:00:38.702[network]         [34mdebug[39m: Dialed discovered peer peer=16...kjGNhe
Feb-10 20:00:38.710[network]         [34mdebug[39m: Dialing discovered peer peer=16...cxEHSd
Feb-10 20:00:38.714[network]         [34mdebug[39m: Dialing discovered peer peer=16...F1LF9s
Feb-10 20:00:38.714[network]         [34mdebug[39m: Dialing discovered peer peer=16...4hYrjs
Feb-10 20:00:38.715[network]         [34mdebug[39m: Dialing discovered peer peer=16...ZE1eoi
Feb-10 20:00:38.716[network]         [34mdebug[39m: Dialing discovered peer peer=16...3zs1SB
Feb-10 20:00:38.717[network]         [34mdebug[39m: Error dialing discovered peer peer=16...uZp5Qz dial with more addresses than allowed
Error: dial with more addresses than allowed
    at DefaultDialer._createDialTarget (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:251:21)
    at DefaultDialer.dial (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:187:20)
    at DefaultConnectionManager.openConnection (file:///usr/app/node_modules/libp2p/src/connection-manager/index.ts:505:26)
    at Libp2pNode.dial (file:///usr/app/node_modules/libp2p/src/libp2p.ts:358:12)
    at PeerDiscovery.dialPeer (file:///usr/app/packages/beacon-node/src/network/peers/discover.ts:458:7)
Feb-10 20:00:38.718[network]         [34mdebug[39m: Dialing discovered peer peer=16...T6beq5
Feb-10 20:00:38.719[network]         [34mdebug[39m: Dialing discovered peer peer=16...7yoyy7
Feb-10 20:00:38.721[network]         [34mdebug[39m: Dialing discovered peer peer=16...cmpgdY
Feb-10 20:00:38.722[network]         [34mdebug[39m: Dialing discovered peer peer=16...dERQ7J
Feb-10 20:00:38.723[network]         [34mdebug[39m: Dialing discovered peer peer=16...PmHJtp
Feb-10 20:00:38.724[network]         [34mdebug[39m: Error dialing discovered peer peer=16...9Sj5HX dial with more addresses than allowed
Error: dial with more addresses than allowed
    at DefaultDialer._createDialTarget (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:251:21)
    at DefaultDialer.dial (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:187:20)
    at DefaultConnectionManager.openConnection (file:///usr/app/node_modules/libp2p/src/connection-manager/index.ts:505:26)
    at Libp2pNode.dial (file:///usr/app/node_modules/libp2p/src/libp2p.ts:358:12)
    at PeerDiscovery.dialPeer (file:///usr/app/packages/beacon-node/src/network/peers/discover.ts:458:7)
Feb-10 20:00:38.725[network]       [36mverbose[39m: peer connected peer=16...PChr7U, direction=outbound, status=OPEN
Feb-10 20:00:38.725[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1634
Feb-10 20:00:38.725[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1635
Feb-10 20:00:38.726[network]       [36mverbose[39m: peer connected peer=16...kjGNhe, direction=outbound, status=OPEN
Feb-10 20:00:38.726[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1636
Feb-10 20:00:38.726[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1637
Feb-10 20:00:38.735[network]         [34mdebug[39m: Dialing discovered peer peer=16...cmpgdY
Feb-10 20:00:38.737[network]         [34mdebug[39m: Dialing discovered peer peer=16...dERQ7J
Feb-10 20:00:38.763[network]         [34mdebug[39m: Dialed discovered peer peer=16...qavN3T
Feb-10 20:00:38.763[network]         [34mdebug[39m: Dialed discovered peer peer=16...qavN3T
Feb-10 20:00:38.777[network]         [34mdebug[39m: Dialing discovered peer peer=16...uT6N3r
Feb-10 20:00:38.779[network]         [34mdebug[39m: Dialed discovered peer peer=16...ccUEXM
Feb-10 20:00:38.779[network]         [34mdebug[39m: Dialing discovered peer peer=16...FCcYay
Feb-10 20:00:38.781[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...ccUEXM, requestId=1638, body=undefined
Feb-10 20:00:38.782[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...ccUEXM, requestId=1638
Feb-10 20:00:38.798[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1637, body=undefined
Feb-10 20:00:38.804[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1636, body=0
Feb-10 20:00:38.806[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1637
Feb-10 20:00:38.806[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1636
Feb-10 20:00:38.807[network]         [34mdebug[39m: Dialing discovered peer peer=16...TvJBRK
Feb-10 20:00:38.808[network]       [36mverbose[39m: peer connected peer=16...qavN3T, direction=outbound, status=OPEN
Feb-10 20:00:38.808[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1639
Feb-10 20:00:38.808[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1640
Feb-10 20:00:38.811[network]       [36mverbose[39m: peer connected peer=16...ccUEXM, direction=outbound, status=OPEN
Feb-10 20:00:38.811[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1641
Feb-10 20:00:38.811[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1642
Feb-10 20:00:38.836[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1634, body=0
Feb-10 20:00:38.837[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1635, body=undefined
Feb-10 20:00:38.838[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1634
Feb-10 20:00:38.838[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1635
Feb-10 20:00:38.853[network]         [34mdebug[39m: Dialing discovered peer peer=16...D1omoC
Feb-10 20:00:38.896[network]         [34mdebug[39m: Irrelevant peer peer=16...kjGNhe, reason=INCOMPATIBLE_FORKS ours: 0x824be431 theirs: 0x56fdb5e0
Feb-10 20:00:38.897[network]         [34mdebug[39m: Req  dialing peer method=goodbye, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1643
Feb-10 20:00:38.898[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1644
Feb-10 20:00:38.899[network]         [34mdebug[39m: Resp received request method=goodbye, client=unknown, peer=16...kjGNhe, requestId=1645, body=1
Feb-10 20:00:38.899[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...kjGNhe, requestId=1646, body=undefined
Feb-10 20:00:38.901[network]       [36mverbose[39m: Resp done method=goodbye, client=unknown, peer=16...kjGNhe, requestId=1645
Feb-10 20:00:38.901[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...kjGNhe, requestId=1646
Feb-10 20:00:38.903[network]       [36mverbose[39m: Received goodbye request peer=16...kjGNhe, goodbye=1, reason=Client shutdown
Feb-10 20:00:38.905[network]       [36mverbose[39m: Req  error method=goodbye, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1643 code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at listOnTimeout (node:internal/timers:538:9)
    at processTimers (node:internal/timers:512:7)
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
Error: REQUEST_ERROR_DIAL_ERROR
    at file:///usr/app/packages/reqresp/src/request/index.ts:118:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at listOnTimeout (node:internal/timers:538:9)
    at processTimers (node:internal/timers:512:7)
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:104:20)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at ReqRespBeaconNode.goodbye (file:///usr/app/packages/beacon-node/src/network/reqresp/ReqRespBeaconNode.ts:144:5)
    at PeerManager.goodbyeAndDisconnect (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:623:7)
Feb-10 20:00:38.905[network]       [36mverbose[39m: Req  error method=metadata, encoding=ssz_snappy, client=undefined, peer=16...kjGNhe, requestId=1644 code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at listOnTimeout (node:internal/timers:538:9)
    at processTimers (node:internal/timers:512:7)
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
Error: REQUEST_ERROR_DIAL_ERROR
    at file:///usr/app/packages/reqresp/src/request/index.ts:118:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at listOnTimeout (node:internal/timers:538:9)
    at processTimers (node:internal/timers:512:7)
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:104:20)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at PeerManager.requestMetadata (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:354:29)
Feb-10 20:00:38.906[network]       [36mverbose[39m: Failed to send goodbye peer=16...kjGNhe method=goodbye, encoding=ssz_snappy, peer=16Uiu2HAm2Q5XMvZ3HFWXjxor7NXWQ45qgNuhqLUatFfzX6kjGNhe, code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at listOnTimeout (node:internal/timers:538:9)
    at processTimers (node:internal/timers:512:7)
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
Error: REQUEST_ERROR_DIAL_ERROR
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:219:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at listOnTimeout (node:internal/timers:538:9)
    at processTimers (node:internal/timers:512:7)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at ReqRespBeaconNode.goodbye (file:///usr/app/packages/beacon-node/src/network/reqresp/ReqRespBeaconNode.ts:144:5)
    at PeerManager.goodbyeAndDisconnect (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:623:7)
Feb-10 20:00:38.948[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1639, body=0
Feb-10 20:00:38.949[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1640, body=undefined
Feb-10 20:00:38.950[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1639
Feb-10 20:00:38.950[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1640
Feb-10 20:00:38.994[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1641, body=0
Feb-10 20:00:38.995[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1642, body=undefined
Feb-10 20:00:38.996[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1641
Feb-10 20:00:38.996[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1642
Feb-10 20:00:38.996[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1647
Feb-10 20:00:38.997[network]       [36mverbose[39m: peer disconnected peer=16...kjGNhe, direction=outbound, status=CLOSING
Feb-10 20:00:38.999[network]         [34mdebug[39m: Dialing discovered peer peer=16...jFyLVa
Feb-10 20:00:39.006[network]         [34mdebug[39m: Dialed discovered peer peer=16...T6beq5
Feb-10 20:00:39.007[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...T6beq5, requestId=1648, body=undefined
Feb-10 20:00:39.008[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...T6beq5, requestId=1648
Feb-10 20:00:39.009[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1649
Feb-10 20:00:39.010[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...qavN3T, requestId=1650, body=undefined
Feb-10 20:00:39.011[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...qavN3T, requestId=1650
Feb-10 20:00:39.013[network]         [34mdebug[39m: Dialed discovered peer peer=16...cmpgdY
Feb-10 20:00:39.013[network]         [34mdebug[39m: Dialed discovered peer peer=16...cmpgdY
Feb-10 20:00:39.047[network]         [34mdebug[39m: Dialed discovered peer peer=16...33BGkB
Feb-10 20:00:39.048[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...33BGkB, requestId=1651, body=undefined
Feb-10 20:00:39.049[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...33BGkB, requestId=1651
Feb-10 20:00:39.081[network]         [34mdebug[39m: Dialed discovered peer peer=16...ZE1eoi
Feb-10 20:00:39.082[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...ZE1eoi, requestId=1652, body=undefined
Feb-10 20:00:39.083[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...ZE1eoi, requestId=1652
Feb-10 20:00:39.084[network]         [34mdebug[39m: Dialed discovered peer peer=16...3zs1SB
Feb-10 20:00:39.085[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...3zs1SB, requestId=1653, body=undefined
Feb-10 20:00:39.086[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...3zs1SB, requestId=1653
Feb-10 20:00:39.092[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1647, body=undefined
Feb-10 20:00:39.093[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...PChr7U, requestId=1647
Feb-10 20:00:39.100[network]         [34mdebug[39m: Dialed discovered peer peer=16...F1LF9s
Feb-10 20:00:39.101[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...F1LF9s, requestId=1655, body=undefined
Feb-10 20:00:39.102[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...F1LF9s, requestId=1655
Feb-10 20:00:39.119[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1656
Feb-10 20:00:39.121[network]         [34mdebug[39m: Dialing discovered peer peer=16...7ZiuSu
Feb-10 20:00:39.124[network]         [34mdebug[39m: Dialing discovered peer peer=16...TvJBRK
Feb-10 20:00:39.127[network]         [34mdebug[39m: Error dialing discovered peer peer=16...D1omoC dial with more addresses than allowed
Error: dial with more addresses than allowed
    at DefaultDialer._createDialTarget (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:251:21)
    at DefaultDialer.dial (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:187:20)
    at DefaultConnectionManager.openConnection (file:///usr/app/node_modules/libp2p/src/connection-manager/index.ts:505:26)
    at Libp2pNode.dial (file:///usr/app/node_modules/libp2p/src/libp2p.ts:358:12)
    at PeerDiscovery.dialPeer (file:///usr/app/packages/beacon-node/src/network/peers/discover.ts:458:7)
Feb-10 20:00:39.140[network]         [34mdebug[39m: Dialed discovered peer peer=16...7yoyy7
Feb-10 20:00:39.141[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...7yoyy7, requestId=1657, body=undefined
Feb-10 20:00:39.142[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...7yoyy7, requestId=1657
Feb-10 20:00:39.142[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1649, body=undefined
Feb-10 20:00:39.143[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...qavN3T, requestId=1649
Feb-10 20:00:39.145[network]         [34mdebug[39m: Dialing discovered peer peer=16...nDFKYm
Feb-10 20:00:39.146[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...7jGdWm, requestId=1654, body=0
Feb-10 20:00:39.147[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...7jGdWm, requestId=1654
Feb-10 20:00:39.152[network]         [34mdebug[39m: Dialed discovered peer peer=16...cxEHSd
Feb-10 20:00:39.186[network]         [34mdebug[39m: Dialing discovered peer peer=16...9YVAmv
Feb-10 20:00:39.196[network]         [34mdebug[39m: Dialed discovered peer peer=16...7S4Fkg
Feb-10 20:00:39.197[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...7S4Fkg, requestId=1658, body=undefined
Feb-10 20:00:39.197[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...7S4Fkg, requestId=1658
Feb-10 20:00:39.198[network]         [34mdebug[39m: Dialing discovered peer peer=16...kd79Uh
Feb-10 20:00:39.198[network]         [34mdebug[39m: Dialing discovered peer peer=16...wTzs1H
Feb-10 20:00:39.199[network]         [34mdebug[39m: Dialing discovered peer peer=16...MSvxvQ
Feb-10 20:00:39.200[network]         [34mdebug[39m: Dialing discovered peer peer=16...sKkyCo
Feb-10 20:00:39.205[network]         [34mdebug[39m: Dialing discovered peer peer=16...4mEL4z
Feb-10 20:00:39.206[network]         [34mdebug[39m: Dialing discovered peer peer=16...FbcLDL
Feb-10 20:00:39.207[network]         [34mdebug[39m: Dialing discovered peer peer=16...G2S8GF
Feb-10 20:00:39.207[network]         [34mdebug[39m: Dialing discovered peer peer=16...15Ev9N
Feb-10 20:00:39.208[network]         [34mdebug[39m: Dialing discovered peer peer=16...Pryg6m
Feb-10 20:00:39.209[network]       [36mverbose[39m: peer connected peer=16...T6beq5, direction=outbound, status=OPEN
Feb-10 20:00:39.209[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1659
Feb-10 20:00:39.209[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1660
Feb-10 20:00:39.211[network]       [36mverbose[39m: peer connected peer=16...cmpgdY, direction=outbound, status=OPEN
Feb-10 20:00:39.211[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1661
Feb-10 20:00:39.212[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1662
Feb-10 20:00:39.213[network]       [36mverbose[39m: peer connected peer=16...33BGkB, direction=outbound, status=OPEN
Feb-10 20:00:39.213[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1663
Feb-10 20:00:39.213[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1664
Feb-10 20:00:39.221[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1656, body=undefined
Feb-10 20:00:39.221[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...ccUEXM, requestId=1656
Feb-10 20:00:39.222[network]         [34mdebug[39m: Dialing discovered peer peer=16...onPW4c
Feb-10 20:00:39.222[network]         [34mdebug[39m: Dialing discovered peer peer=16...XACeQ3
Feb-10 20:00:39.223[network]       [36mverbose[39m: peer connected peer=16...ZE1eoi, direction=outbound, status=OPEN
Feb-10 20:00:39.223[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1665
Feb-10 20:00:39.223[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1666
Feb-10 20:00:39.224[network]       [36mverbose[39m: peer connected peer=16...3zs1SB, direction=outbound, status=OPEN
Feb-10 20:00:39.224[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1667
Feb-10 20:00:39.225[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1668
Feb-10 20:00:39.228[network]       [36mverbose[39m: peer connected peer=16...F1LF9s, direction=outbound, status=OPEN
Feb-10 20:00:39.228[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1669
Feb-10 20:00:39.228[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1670
Feb-10 20:00:39.230[network]         [34mdebug[39m: Dialing discovered peer peer=16...7ZiuSu
Feb-10 20:00:39.233[network]       [36mverbose[39m: peer connected peer=16...7yoyy7, direction=outbound, status=OPEN
Feb-10 20:00:39.233[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1671
Feb-10 20:00:39.233[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1672
Feb-10 20:00:39.250[network]         [34mdebug[39m: Dialed discovered peer peer=16...FCcYay
Feb-10 20:00:39.251[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...FCcYay, requestId=1673, body=undefined
Feb-10 20:00:39.251[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...FCcYay, requestId=1673
Feb-10 20:00:39.258[network]       [36mverbose[39m: peer connected peer=16...cxEHSd, direction=outbound, status=OPEN
Feb-10 20:00:39.258[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1674
Feb-10 20:00:39.258[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1675
Feb-10 20:00:39.260[network]         [34mdebug[39m: Error dialing discovered peer peer=16...9YVAmv dial with more addresses than allowed
Error: dial with more addresses than allowed
    at DefaultDialer._createDialTarget (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:251:21)
    at DefaultDialer.dial (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:187:20)
    at DefaultConnectionManager.openConnection (file:///usr/app/node_modules/libp2p/src/connection-manager/index.ts:505:26)
    at Libp2pNode.dial (file:///usr/app/node_modules/libp2p/src/libp2p.ts:358:12)
    at PeerDiscovery.dialPeer (file:///usr/app/packages/beacon-node/src/network/peers/discover.ts:458:7)
Feb-10 20:00:39.263[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1659, body=0
Feb-10 20:00:39.263[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1660, body=undefined
Feb-10 20:00:39.264[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1659
Feb-10 20:00:39.264[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1660
Feb-10 20:00:39.265[network]         [34mdebug[39m: Dialing discovered peer peer=16...byYxNN
Feb-10 20:00:39.265[network]       [36mverbose[39m: peer connected peer=16...7S4Fkg, direction=outbound, status=OPEN
Feb-10 20:00:39.265[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1676
Feb-10 20:00:39.265[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1677
Feb-10 20:00:39.271[network]         [34mdebug[39m: Error dialing discovered peer peer=16...kd79Uh dial with more addresses than allowed
Error: dial with more addresses than allowed
    at DefaultDialer._createDialTarget (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:251:21)
    at DefaultDialer.dial (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:187:20)
    at DefaultConnectionManager.openConnection (file:///usr/app/node_modules/libp2p/src/connection-manager/index.ts:505:26)
    at Libp2pNode.dial (file:///usr/app/node_modules/libp2p/src/libp2p.ts:358:12)
    at PeerDiscovery.dialPeer (file:///usr/app/packages/beacon-node/src/network/peers/discover.ts:458:7)
Feb-10 20:00:39.273[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1661, body=0
Feb-10 20:00:39.274[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1662, body=undefined
Feb-10 20:00:39.275[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1661
Feb-10 20:00:39.275[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1662
Feb-10 20:00:39.285[network]         [34mdebug[39m: Error dialing discovered peer peer=16...15Ev9N dial with more addresses than allowed
Error: dial with more addresses than allowed
    at DefaultDialer._createDialTarget (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:251:21)
    at DefaultDialer.dial (file:///usr/app/node_modules/libp2p/src/connection-manager/dialer/index.ts:187:20)
    at DefaultConnectionManager.openConnection (file:///usr/app/node_modules/libp2p/src/connection-manager/index.ts:505:26)
    at Libp2pNode.dial (file:///usr/app/node_modules/libp2p/src/libp2p.ts:358:12)
    at PeerDiscovery.dialPeer (file:///usr/app/packages/beacon-node/src/network/peers/discover.ts:458:7)
Feb-10 20:00:39.299[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1665, body=0
Feb-10 20:00:39.301[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1666, body=undefined
Feb-10 20:00:39.302[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1665
Feb-10 20:00:39.302[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1666
Feb-10 20:00:39.304[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1667, body=0
Feb-10 20:00:39.305[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1668, body=undefined
Feb-10 20:00:39.305[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1667
Feb-10 20:00:39.306[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1668
Feb-10 20:00:39.308[network]         [34mdebug[39m: Dialing discovered peer peer=16...hm5MfA
Feb-10 20:00:39.309[network]       [36mverbose[39m: peer connected peer=16...FCcYay, direction=outbound, status=OPEN
Feb-10 20:00:39.309[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1678
Feb-10 20:00:39.309[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1679
Feb-10 20:00:39.318[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1671, body=0
Feb-10 20:00:39.319[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1672, body=undefined
Feb-10 20:00:39.320[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1671
Feb-10 20:00:39.320[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1672
Feb-10 20:00:39.332[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1674, body=0
Feb-10 20:00:39.332[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1675, body=undefined
Feb-10 20:00:39.333[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1674
Feb-10 20:00:39.333[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1675
Feb-10 20:00:39.355[network]         [34mdebug[39m: Dialed discovered peer peer=16...uT6N3r
Feb-10 20:00:39.356[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...uT6N3r, requestId=1680, body=undefined
Feb-10 20:00:39.357[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...uT6N3r, requestId=1680
Feb-10 20:00:39.358[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1663, body=0
Feb-10 20:00:39.358[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1664, body=undefined
Feb-10 20:00:39.360[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1663
Feb-10 20:00:39.360[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1664
Feb-10 20:00:39.372[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1681
Feb-10 20:00:39.379[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1682
Feb-10 20:00:39.381[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...cmpgdY, requestId=1683, body=undefined
Feb-10 20:00:39.382[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...cmpgdY, requestId=1683
Feb-10 20:00:39.401[network]         [34mdebug[39m: Dialed discovered peer peer=16...4hYrjs
Feb-10 20:00:39.402[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...4hYrjs, requestId=1684, body=undefined
Feb-10 20:00:39.403[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...4hYrjs, requestId=1684
Feb-10 20:00:39.404[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1669, body=0
Feb-10 20:00:39.405[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1670, body=undefined
Feb-10 20:00:39.405[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1669
Feb-10 20:00:39.406[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1670
Feb-10 20:00:39.406[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1685
Feb-10 20:00:39.407[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1686
Feb-10 20:00:39.419[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1678, body=0
Feb-10 20:00:39.419[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1679, body=undefined
Feb-10 20:00:39.420[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1678
Feb-10 20:00:39.420[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1679
Feb-10 20:00:39.421[network]         [34mdebug[39m: Dialing discovered peer peer=16...wQSREX
Feb-10 20:00:39.457[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1687
Feb-10 20:00:39.468[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1688
Feb-10 20:00:39.470[network]         [34mdebug[39m: Dialed discovered peer peer=16...TvJBRK
Feb-10 20:00:39.470[network]         [34mdebug[39m: Dialed discovered peer peer=16...TvJBRK
Feb-10 20:00:39.478[network]         [34mdebug[39m: Dialed discovered peer peer=16...sKkyCo
Feb-10 20:00:39.479[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...sKkyCo, requestId=1690, body=undefined
Feb-10 20:00:39.480[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...sKkyCo, requestId=1690
Feb-10 20:00:39.517[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1691
Feb-10 20:00:39.519[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...cxEHSd, requestId=1692, body=undefined
Feb-10 20:00:39.520[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...cxEHSd, requestId=1692
Feb-10 20:00:39.523[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1685, body=undefined
Feb-10 20:00:39.523[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...ZE1eoi, requestId=1685
Feb-10 20:00:39.524[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1676, body=0
Feb-10 20:00:39.525[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1677, body=undefined
Feb-10 20:00:39.526[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1676
Feb-10 20:00:39.526[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1677
Feb-10 20:00:39.528[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1682, body=undefined
Feb-10 20:00:39.528[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...cmpgdY, requestId=1682
Feb-10 20:00:39.529[network]         [34mdebug[39m: Dialing discovered peer peer=16...1EZ3cU
Feb-10 20:00:39.546[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1681, body=undefined
Feb-10 20:00:39.547[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...T6beq5, requestId=1681
Feb-10 20:00:39.577[network]         [34mdebug[39m: Dialed discovered peer peer=16...MSvxvQ
Feb-10 20:00:39.578[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...MSvxvQ, requestId=1694, body=undefined
Feb-10 20:00:39.579[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...MSvxvQ, requestId=1694
Feb-10 20:00:39.580[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1686, body=undefined
Feb-10 20:00:39.580[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...3zs1SB, requestId=1686
Feb-10 20:00:39.582[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1695
Feb-10 20:00:39.593[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1696
Feb-10 20:00:39.611[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1687, body=undefined
Feb-10 20:00:39.611[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...33BGkB, requestId=1687
Feb-10 20:00:39.622[network]         [34mdebug[39m: Dialed discovered peer peer=16...XACeQ3
Feb-10 20:00:39.623[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...XACeQ3, requestId=1698, body=undefined
Feb-10 20:00:39.624[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...XACeQ3, requestId=1698
Feb-10 20:00:39.631[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1688, body=undefined
Feb-10 20:00:39.631[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...7yoyy7, requestId=1688
Feb-10 20:00:39.634[network]         [34mdebug[39m: Dialed discovered peer peer=16...dERQ7J
Feb-10 20:00:39.634[network]         [34mdebug[39m: Dialed discovered peer peer=16...dERQ7J
Feb-10 20:00:39.648[network]       [36mverbose[39m: peer connected peer=16...uT6N3r, direction=outbound, status=OPEN
Feb-10 20:00:39.648[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1699
Feb-10 20:00:39.649[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1700
Feb-10 20:00:39.652[network]       [36mverbose[39m: cannot tag peer peerId=16Uiu2HAmSoyaAGQ7JKYT7UV7URdgdkQc12tjxKaiSdPjfDT6beq5 Peer already tagged
Error: Peer already tagged
    at PersistentPeerStore.tagPeer (file:///usr/app/node_modules/@libp2p/peer-store/src/index.ts:140:23)
Feb-10 20:00:39.657[network]       [36mverbose[39m: peer connected peer=16...4hYrjs, direction=outbound, status=OPEN
Feb-10 20:00:39.657[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1701
Feb-10 20:00:39.657[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1702
Feb-10 20:00:39.661[network]         [34mdebug[39m: Dialed discovered peer peer=16...7ZiuSu
Feb-10 20:00:39.661[network]         [34mdebug[39m: Dialed discovered peer peer=16...7ZiuSu
Feb-10 20:00:39.662[network]         [34mdebug[39m: Resp received request method=goodbye, client=unknown, peer=16...TvJBRK, requestId=1689, body=129
Feb-10 20:00:39.663[network]       [36mverbose[39m: Resp done method=goodbye, client=unknown, peer=16...TvJBRK, requestId=1689
Feb-10 20:00:39.664[network]         [34mdebug[39m: Dialed discovered peer peer=16...FbcLDL
Feb-10 20:00:39.666[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...FbcLDL, requestId=1703, body=undefined
Feb-10 20:00:39.667[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...FbcLDL, requestId=1703
Feb-10 20:00:39.672[network]         [34mdebug[39m: Dialed discovered peer peer=16...4mEL4z
Feb-10 20:00:39.674[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...4mEL4z, requestId=1704, body=undefined
Feb-10 20:00:39.674[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...4mEL4z, requestId=1704
Feb-10 20:00:39.675[network]       [36mverbose[39m: Received goodbye request peer=16...TvJBRK, goodbye=129, reason=Client has too many peers
Feb-10 20:00:39.677[network]       [36mverbose[39m: cannot tag peer peerId=16Uiu2HAm2D9SPyFT8oKFdT3CJ6oDDkMp6VBgyjmfoajNkQZE1eoi Peer already tagged
Error: Peer already tagged
    at PersistentPeerStore.tagPeer (file:///usr/app/node_modules/@libp2p/peer-store/src/index.ts:140:23)
Feb-10 20:00:39.682[network]       [36mverbose[39m: peer connected peer=16...TvJBRK, direction=outbound, status=CLOSING
Feb-10 20:00:39.683[network]       [36mverbose[39m: peer connected peer=16...sKkyCo, direction=outbound, status=OPEN
Feb-10 20:00:39.683[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1705
Feb-10 20:00:39.683[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1706
Feb-10 20:00:39.705[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1691, body=undefined
Feb-10 20:00:39.705[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...cxEHSd, requestId=1691
Feb-10 20:00:39.707[network]         [34mdebug[39m: Dialed discovered peer peer=16...G2S8GF
Feb-10 20:00:39.708[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...G2S8GF, requestId=1707, body=undefined
Feb-10 20:00:39.709[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...G2S8GF, requestId=1707
Feb-10 20:00:39.710[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...XfKg5C, requestId=1693, body=0
Feb-10 20:00:39.710[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...XfKg5C, requestId=1693
Feb-10 20:00:39.711[network]         [34mdebug[39m: Dialed discovered peer peer=16...Pryg6m
Feb-10 20:00:39.712[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...Pryg6m, requestId=1708, body=undefined
Feb-10 20:00:39.713[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...Pryg6m, requestId=1708
Feb-10 20:00:39.714[network]       [36mverbose[39m: peer disconnected peer=16...TvJBRK, direction=outbound, status=CLOSING
Feb-10 20:00:39.716[network]       [36mverbose[39m: peer connected peer=16...MSvxvQ, direction=outbound, status=OPEN
Feb-10 20:00:39.716[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1709
Feb-10 20:00:39.716[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1710
Feb-10 20:00:39.719[network]       [36mverbose[39m: cannot tag peer peerId=16Uiu2HAkwfGDDXDwAhLkjZ5g8RCi7K82JgKztyLRr6cHfaF1LF9s Peer already tagged
Error: Peer already tagged
    at PersistentPeerStore.tagPeer (file:///usr/app/node_modules/@libp2p/peer-store/src/index.ts:140:23)
Feb-10 20:00:39.722[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...W5xRbh, requestId=1697, body=0
Feb-10 20:00:39.723[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...W5xRbh, requestId=1697
Feb-10 20:00:39.732[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1695, body=undefined
Feb-10 20:00:39.732[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...F1LF9s, requestId=1695
Feb-10 20:00:39.733[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1696, body=undefined
Feb-10 20:00:39.734[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...FCcYay, requestId=1696
Feb-10 20:00:39.735[network]         [34mdebug[39m: Dialed discovered peer peer=16...wRz8C6
Feb-10 20:00:39.736[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...wRz8C6, requestId=1711, body=undefined
Feb-10 20:00:39.737[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...wRz8C6, requestId=1711
Feb-10 20:00:39.751[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1705, body=0
Feb-10 20:00:39.752[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1706, body=undefined
Feb-10 20:00:39.754[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1705
Feb-10 20:00:39.754[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1706
Feb-10 20:00:39.756[network]         [34mdebug[39m: Dialed discovered peer peer=16...jFyLVa
Feb-10 20:00:39.758[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...jFyLVa, requestId=1712, body=undefined
Feb-10 20:00:39.759[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...jFyLVa, requestId=1712
Feb-10 20:00:39.764[network]         [34mdebug[39m: Dialed discovered peer peer=16...nDFKYm
Feb-10 20:00:39.765[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...nDFKYm, requestId=1713, body=undefined
Feb-10 20:00:39.766[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...nDFKYm, requestId=1713
Feb-10 20:00:39.767[network]       [36mverbose[39m: peer connected peer=16...XACeQ3, direction=outbound, status=OPEN
Feb-10 20:00:39.767[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1714
Feb-10 20:00:39.767[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1715
Feb-10 20:00:39.770[network]       [36mverbose[39m: peer connected peer=16...dERQ7J, direction=outbound, status=OPEN
Feb-10 20:00:39.770[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1716
Feb-10 20:00:39.770[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1717
Feb-10 20:00:39.779[network]         [34mdebug[39m: Dialed discovered peer peer=16...PmHJtp
Feb-10 20:00:39.780[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...PmHJtp, requestId=1718, body=undefined
Feb-10 20:00:39.781[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...PmHJtp, requestId=1718
Feb-10 20:00:39.781[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1719
Feb-10 20:00:39.783[network]         [34mdebug[39m: Dialed discovered peer peer=16...wTzs1H
Feb-10 20:00:39.784[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...wTzs1H, requestId=1720, body=undefined
Feb-10 20:00:39.785[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...wTzs1H, requestId=1720
Feb-10 20:00:39.786[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1709, body=0
Feb-10 20:00:39.787[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1710, body=undefined
Feb-10 20:00:39.787[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1709
Feb-10 20:00:39.788[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1710
Feb-10 20:00:39.814[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1722
Feb-10 20:00:39.816[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1699, body=0
Feb-10 20:00:39.817[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1700, body=undefined
Feb-10 20:00:39.818[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1699
Feb-10 20:00:39.818[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1700
Feb-10 20:00:39.832[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1701, body=0
Feb-10 20:00:39.833[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1702, body=undefined
Feb-10 20:00:39.834[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1701
Feb-10 20:00:39.834[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1702
Feb-10 20:00:39.863[network]         [34mdebug[39m: Dialed discovered peer peer=16...hm5MfA
Feb-10 20:00:39.864[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1723
Feb-10 20:00:39.870[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1714, body=0
Feb-10 20:00:39.871[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1715, body=undefined
Feb-10 20:00:39.872[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1714
Feb-10 20:00:39.872[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1715
Feb-10 20:00:39.897[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1722, body=undefined
Feb-10 20:00:39.898[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...sKkyCo, requestId=1722
Feb-10 20:00:39.903[network]         [34mdebug[39m: Dialed discovered peer peer=16...byYxNN
Feb-10 20:00:39.906[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...byYxNN, requestId=1724, body=undefined
Feb-10 20:00:39.907[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...byYxNN, requestId=1724
Feb-10 20:00:39.941[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1716, body=0
Feb-10 20:00:39.942[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1717, body=undefined
Feb-10 20:00:39.942[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1716
Feb-10 20:00:39.943[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1717
Feb-10 20:00:39.953[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1725
Feb-10 20:00:39.954[network]         [34mdebug[39m: Dialed discovered peer peer=16...1EZ3cU
Feb-10 20:00:39.956[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...1EZ3cU, requestId=1726, body=undefined
Feb-10 20:00:39.956[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...1EZ3cU, requestId=1726
Feb-10 20:00:39.958[network]       [36mverbose[39m: cannot tag peer peerId=16Uiu2HAm75k5NbunDMa7HEtwwU7n7xAcjr4ByZBUvBvfejXACeQ3 Peer already tagged
Error: Peer already tagged
    at PersistentPeerStore.tagPeer (file:///usr/app/node_modules/@libp2p/peer-store/src/index.ts:140:23)
Feb-10 20:00:39.959[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1723, body=undefined
Feb-10 20:00:39.959[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...MSvxvQ, requestId=1723
Feb-10 20:00:39.960[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1727
Feb-10 20:00:39.961[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1719, body=undefined
Feb-10 20:00:39.961[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...7S4Fkg, requestId=1719
Feb-10 20:00:39.996[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1728
Feb-10 20:00:40.005[chain]         [36mverbose[39m: Clock slot slot=7412460
Feb-10 20:00:40.033[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1725, body=undefined
Feb-10 20:00:40.033[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...XACeQ3, requestId=1725
Feb-10 20:00:40.044[network]         [34mdebug[39m: Resp received request method=ping, client=unknown, peer=16...vNHDMF, requestId=1721, body=80409
Feb-10 20:00:40.044[network]       [36mverbose[39m: Resp done method=ping, client=unknown, peer=16...vNHDMF, requestId=1721
Feb-10 20:00:40.125[network]       [36mverbose[39m: Req  done method=ping, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1716
Feb-10 20:00:40.125[network]       [36mverbose[39m: Req  done method=status, encoding=ssz_snappy, client=undefined, peer=16...dERQ7J, requestId=1717
Feb-10 20:00:40.127[network]       [36mverbose[39m: Req  error method=metadata, encoding=ssz_snappy, client=undefined, peer=16...uT6N3r, requestId=1727 code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
    at read (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:78:15)
    at Module.readString (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:97:15)
    at Module.select (file:///usr/app/node_modules/@libp2p/multistream-select/src/select.ts:76:18)
    at ConnectionImpl.newStream [as _newStream] (file:///usr/app/node_modules/libp2p/src/upgrader.ts:445:40)
Error: REQUEST_ERROR_DIAL_ERROR
    at file:///usr/app/packages/reqresp/src/request/index.ts:118:15
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:104:20)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at PeerManager.requestMetadata (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:354:29)
Feb-10 20:00:40.131[network]       [36mverbose[39m: Req  error method=metadata, encoding=ssz_snappy, client=undefined, peer=16...4hYrjs, requestId=1728 code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
    at read (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:78:15)
    at Module.readString (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:97:15)
    at Module.select (file:///usr/app/node_modules/@libp2p/multistream-select/src/select.ts:76:18)
    at ConnectionImpl.newStream [as _newStream] (file:///usr/app/node_modules/libp2p/src/upgrader.ts:445:40)
Error: REQUEST_ERROR_DIAL_ERROR
    at file:///usr/app/packages/reqresp/src/request/index.ts:118:15
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:104:20)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at PeerManager.requestMetadata (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:354:29)
Feb-10 20:00:40.150[network]       [36mverbose[39m: peer disconnected peer=16...FfGQ5M, direction=outbound, status=CLOSING
Feb-10 20:00:40.150[network]       [36mverbose[39m: peer disconnected peer=16...LUDWEv, direction=outbound, status=CLOSING
Feb-10 20:00:40.151[network]       [36mverbose[39m: peer disconnected peer=16...JEWMCj, direction=outbound, status=CLOSING
Feb-10 20:00:40.151[network]       [36mverbose[39m: peer disconnected peer=16...LYYtqn, direction=outbound, status=CLOSING
Feb-10 20:00:40.151[network]       [36mverbose[39m: peer disconnected peer=16...EZLaj4, direction=outbound, status=CLOSING
Feb-10 20:00:40.152[network]       [36mverbose[39m: peer disconnected peer=16...NQNTGi, direction=outbound, status=CLOSING
Feb-10 20:00:40.152[network]       [36mverbose[39m: peer disconnected peer=16...cEh4hM, direction=outbound, status=CLOSING
Feb-10 20:00:40.153[network]       [36mverbose[39m: peer disconnected peer=16...XfKg5C, direction=outbound, status=CLOSING
Feb-10 20:00:40.154[network]       [36mverbose[39m: peer disconnected peer=16...W5xRbh, direction=outbound, status=CLOSING
Feb-10 20:00:40.154[network]       [36mverbose[39m: peer disconnected peer=16...V3hMhX, direction=outbound, status=CLOSING
Feb-10 20:00:40.154[network]       [36mverbose[39m: peer disconnected peer=16...cXcVeC, direction=outbound, status=CLOSING
Feb-10 20:00:40.154[network]       [36mverbose[39m: peer disconnected peer=16...pQoxUB, direction=outbound, status=CLOSING
Feb-10 20:00:40.155[network]       [36mverbose[39m: peer disconnected peer=16...MW8xf1, direction=outbound, status=CLOSING
Feb-10 20:00:40.155[network]       [36mverbose[39m: peer disconnected peer=16...fM3bvA, direction=outbound, status=CLOSING
Feb-10 20:00:40.155[network]       [36mverbose[39m: peer disconnected peer=16...S2uzTe, direction=outbound, status=CLOSING
Feb-10 20:00:40.156[network]       [36mverbose[39m: peer disconnected peer=16...t6XGxA, direction=outbound, status=CLOSING
Feb-10 20:00:40.156[network]       [36mverbose[39m: peer disconnected peer=16...vNHDMF, direction=outbound, status=CLOSING
Feb-10 20:00:40.156[network]       [36mverbose[39m: peer disconnected peer=16...m5xaX2, direction=outbound, status=CLOSING
Feb-10 20:00:40.156[network]       [36mverbose[39m: peer disconnected peer=16...PNA5Xm, direction=outbound, status=CLOSING
Feb-10 20:00:40.157[network]       [36mverbose[39m: peer disconnected peer=16...TfvNac, direction=outbound, status=CLOSING
Feb-10 20:00:40.157[network]       [36mverbose[39m: peer disconnected peer=16...pdwzti, direction=outbound, status=CLOSING
Feb-10 20:00:40.157[network]       [36mverbose[39m: peer disconnected peer=16...7jGdWm, direction=outbound, status=CLOSING
Feb-10 20:00:40.158[network]       [36mverbose[39m: peer disconnected peer=16...SryPHV, direction=outbound, status=CLOSING
Feb-10 20:00:40.158[network]       [36mverbose[39m: peer disconnected peer=16...EaSdma, direction=outbound, status=CLOSING
Feb-10 20:00:40.158[network]       [36mverbose[39m: peer disconnected peer=16...WbA5vc, direction=outbound, status=CLOSING
Feb-10 20:00:40.158[network]       [36mverbose[39m: peer disconnected peer=16...7hZxNm, direction=outbound, status=CLOSING
Feb-10 20:00:40.159[network]       [36mverbose[39m: peer disconnected peer=16...g1RX9L, direction=outbound, status=CLOSING
Feb-10 20:00:40.159[network]       [36mverbose[39m: peer disconnected peer=16...VwosZw, direction=outbound, status=CLOSING
Feb-10 20:00:40.159[network]       [36mverbose[39m: peer disconnected peer=16...PChr7U, direction=outbound, status=CLOSING
Feb-10 20:00:40.159[network]       [36mverbose[39m: peer disconnected peer=16...qavN3T, direction=outbound, status=CLOSING
Feb-10 20:00:40.160[network]       [36mverbose[39m: peer disconnected peer=16...ccUEXM, direction=outbound, status=CLOSING
Feb-10 20:00:40.160[network]       [36mverbose[39m: peer disconnected peer=16...7S4Fkg, direction=outbound, status=CLOSING
Feb-10 20:00:40.160[network]       [36mverbose[39m: peer disconnected peer=16...33BGkB, direction=outbound, status=CLOSING
Feb-10 20:00:40.161[network]       [36mverbose[39m: peer disconnected peer=16...3zs1SB, direction=outbound, status=CLOSING
Feb-10 20:00:40.161[network]       [36mverbose[39m: peer disconnected peer=16...ZE1eoi, direction=outbound, status=CLOSING
Feb-10 20:00:40.161[network]       [36mverbose[39m: peer disconnected peer=16...cmpgdY, direction=outbound, status=CLOSING
Feb-10 20:00:40.161[network]       [36mverbose[39m: peer disconnected peer=16...T6beq5, direction=outbound, status=CLOSING
Feb-10 20:00:40.162[network]       [36mverbose[39m: peer disconnected peer=16...F1LF9s, direction=outbound, status=CLOSING
Feb-10 20:00:40.162[network]       [36mverbose[39m: peer disconnected peer=16...cxEHSd, direction=outbound, status=CLOSING
Feb-10 20:00:40.166[network]       [36mverbose[39m: peer disconnected peer=16...7yoyy7, direction=outbound, status=CLOSING
Feb-10 20:00:40.167[network]       [36mverbose[39m: peer disconnected peer=16...FCcYay, direction=outbound, status=CLOSING
Feb-10 20:00:40.167[network]       [36mverbose[39m: peer disconnected peer=16...4hYrjs, direction=outbound, status=CLOSING
Feb-10 20:00:40.167[network]       [36mverbose[39m: peer disconnected peer=16...uT6N3r, direction=outbound, status=CLOSING
Feb-10 20:00:40.167[network]       [36mverbose[39m: peer disconnected peer=16...sKkyCo, direction=outbound, status=CLOSING
Feb-10 20:00:40.168[network]       [36mverbose[39m: peer disconnected peer=16...MSvxvQ, direction=outbound, status=CLOSING
Feb-10 20:00:40.168[network]       [36mverbose[39m: peer disconnected peer=16...XACeQ3, direction=outbound, status=CLOSING
Feb-10 20:00:40.168[network]       [36mverbose[39m: peer disconnected peer=16...FbcLDL, direction=outbound, status=CLOSING
Feb-10 20:00:40.169[network]       [36mverbose[39m: peer disconnected peer=16...wRz8C6, direction=outbound, status=CLOSING
Feb-10 20:00:40.169[network]       [36mverbose[39m: peer disconnected peer=16...dERQ7J, direction=outbound, status=CLOSING
Feb-10 20:00:40.169[network]       [36mverbose[39m: peer disconnected peer=16...PmHJtp, direction=outbound, status=CLOSING
Feb-10 20:00:40.169[network]       [36mverbose[39m: peer disconnected peer=16...jFyLVa, direction=outbound, status=CLOSING
Feb-10 20:00:40.170[network]       [36mverbose[39m: peer disconnected peer=16...nDFKYm, direction=outbound, status=CLOSING
Feb-10 20:00:40.170[network]       [36mverbose[39m: peer disconnected peer=16...7ZiuSu, direction=outbound, status=CLOSING
Feb-10 20:00:40.170[network]       [36mverbose[39m: peer disconnected peer=16...4mEL4z, direction=outbound, status=CLOSING
Feb-10 20:00:40.170[network]       [36mverbose[39m: peer disconnected peer=16...G2S8GF, direction=outbound, status=CLOSING
Feb-10 20:00:40.171[network]       [36mverbose[39m: peer disconnected peer=16...wTzs1H, direction=outbound, status=CLOSING
Feb-10 20:00:40.171[network]       [36mverbose[39m: peer disconnected peer=16...Pryg6m, direction=outbound, status=CLOSING
Feb-10 20:00:40.171[network]       [36mverbose[39m: peer connected peer=16...G2S8GF, direction=outbound, status=CLOSED
Feb-10 20:00:40.173[network]       [36mverbose[39m: peer connected peer=16...nDFKYm, direction=outbound, status=CLOSED
Feb-10 20:00:40.174[network]       [36mverbose[39m: peer connected peer=16...4mEL4z, direction=outbound, status=CLOSED
Feb-10 20:00:40.175[network]       [36mverbose[39m: peer connected peer=16...7ZiuSu, direction=outbound, status=CLOSED
Feb-10 20:00:40.176[network]       [36mverbose[39m: peer connected peer=16...PmHJtp, direction=outbound, status=CLOSED
Feb-10 20:00:40.177[network]       [36mverbose[39m: peer connected peer=16...wTzs1H, direction=outbound, status=CLOSED
Feb-10 20:00:40.179[network]       [36mverbose[39m: peer connected peer=16...jFyLVa, direction=outbound, status=CLOSED
Feb-10 20:00:40.180[network]       [36mverbose[39m: peer connected peer=16...wRz8C6, direction=outbound, status=CLOSED
Feb-10 20:00:40.180[network]       [36mverbose[39m: peer connected peer=16...Pryg6m, direction=outbound, status=CLOSED
Feb-10 20:00:40.181[network]       [36mverbose[39m: peer connected peer=16...FbcLDL, direction=outbound, status=CLOSED
Feb-10 20:00:40.204[network]       [36mverbose[39m: peer disconnected peer=16...hm5MfA, direction=outbound, status=CLOSING
Feb-10 20:00:40.204[network]       [36mverbose[39m: peer connected peer=16...hm5MfA, direction=outbound, status=CLOSED
Feb-10 20:00:40.239[network]         [34mdebug[39m: Dialed discovered peer peer=16...wQSREX
Feb-10 20:00:40.240[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...wQSREX, requestId=1729, body=undefined
Feb-10 20:00:40.242[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...wQSREX, requestId=1729
Feb-10 20:00:40.262[network]       [36mverbose[39m: peer disconnected peer=16...byYxNN, direction=outbound, status=CLOSING
Feb-10 20:00:40.262[network]       [36mverbose[39m: peer connected peer=16...byYxNN, direction=outbound, status=CLOSED
Feb-10 20:00:40.267[network]       [36mverbose[39m: peer connected peer=16...wQSREX, direction=outbound, status=OPEN
Feb-10 20:00:40.267[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1730
Feb-10 20:00:40.267[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1731
Feb-10 20:00:40.272[network]       [36mverbose[39m: peer disconnected peer=16...1EZ3cU, direction=outbound, status=CLOSING
Feb-10 20:00:40.273[network]       [36mverbose[39m: peer connected peer=16...1EZ3cU, direction=outbound, status=CLOSED
Feb-10 20:00:40.305[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23691576, toBlock=23692575
Feb-10 20:00:40.320[network]         [34mdebug[39m: Dialing discovered peer peer=16...hm5MfA
Feb-10 20:00:40.608[network]         [34mdebug[39m: Dialing discovered peer peer=16...cJj6Nw
Feb-10 20:00:40.610[network]         [34mdebug[39m: Dialing discovered peer peer=16...cJj6Nw
Feb-10 20:00:40.624[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1730, body=0
Feb-10 20:00:40.625[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1731, body=undefined
Feb-10 20:00:40.631[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1730
Feb-10 20:00:40.631[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1731
Feb-10 20:00:40.639[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23692576, toBlock=23693575
Feb-10 20:00:40.717[network]       [36mverbose[39m: peer connected peer=16...hm5MfA, direction=outbound, status=OPEN
Feb-10 20:00:40.718[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1732
Feb-10 20:00:40.718[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1733
Feb-10 20:00:40.719[network]         [34mdebug[39m: Dialed discovered peer peer=16...hm5MfA
Feb-10 20:00:40.803[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1733, body=undefined
Feb-10 20:00:40.805[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1732, body=0
Feb-10 20:00:40.812[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1733
Feb-10 20:00:40.812[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1732
Feb-10 20:00:40.828[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1734
Feb-10 20:00:40.886[network]         [34mdebug[39m: Req  dialing peer method=metadata, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1735
Feb-10 20:00:40.887[network]         [34mdebug[39m: Irrelevant peer peer=16...hm5MfA, reason=INCOMPATIBLE_FORKS ours: 0x824be431 theirs: 0x56fdb5e0
Feb-10 20:00:40.887[network]         [34mdebug[39m: Req  dialing peer method=goodbye, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1736
Feb-10 20:00:40.889[network]         [34mdebug[39m: Resp received request method=metadata, client=unknown, peer=16...hm5MfA, requestId=1737, body=undefined
Feb-10 20:00:40.891[network]         [34mdebug[39m: Resp received request method=goodbye, client=unknown, peer=16...hm5MfA, requestId=1738, body=1
Feb-10 20:00:40.891[network]       [36mverbose[39m: Resp done method=metadata, client=unknown, peer=16...hm5MfA, requestId=1737
Feb-10 20:00:40.892[network]       [36mverbose[39m: Resp done method=goodbye, client=unknown, peer=16...hm5MfA, requestId=1738
Feb-10 20:00:40.892[network]       [36mverbose[39m: Received goodbye request peer=16...hm5MfA, goodbye=1, reason=Client shutdown
Feb-10 20:00:40.894[network]       [36mverbose[39m: Req  error method=metadata, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1735 code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
    at read (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:78:15)
    at Module.readString (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:97:15)
    at Module.select (file:///usr/app/node_modules/@libp2p/multistream-select/src/select.ts:76:18)
Error: REQUEST_ERROR_DIAL_ERROR
    at file:///usr/app/packages/reqresp/src/request/index.ts:118:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:104:20)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at PeerManager.requestMetadata (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:354:29)
Feb-10 20:00:40.894[network]       [36mverbose[39m: Req  error method=goodbye, encoding=ssz_snappy, client=undefined, peer=16...hm5MfA, requestId=1736 code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
    at read (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:78:15)
    at Module.readString (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:97:15)
    at Module.select (file:///usr/app/node_modules/@libp2p/multistream-select/src/select.ts:76:18)
Error: REQUEST_ERROR_DIAL_ERROR
    at file:///usr/app/packages/reqresp/src/request/index.ts:118:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:104:20)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at ReqRespBeaconNode.goodbye (file:///usr/app/packages/beacon-node/src/network/reqresp/ReqRespBeaconNode.ts:144:5)
    at PeerManager.goodbyeAndDisconnect (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:623:7)
Feb-10 20:00:40.895[network]       [36mverbose[39m: Failed to send goodbye peer=16...hm5MfA method=goodbye, encoding=ssz_snappy, peer=16Uiu2HAkvhXgCaW8Gmye1gUhswsKAcwqHwafU6ZwhRSjmxhm5MfA, code=REQUEST_ERROR_DIAL_ERROR, error=stream ended before 1 bytes became available
Error: stream ended before 1 bytes became available
    at file:///usr/app/node_modules/it-reader/src/index.ts:44:9
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.next (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:62:23)
    at abortable (file:///usr/app/node_modules/abortable-iterator/src/index.ts:42:18)
    at decoder (file:///usr/app/node_modules/it-length-prefixed/src/decode.ts:58:22)
    at first (file:///usr/app/node_modules/it-first/src/index.ts:7:20)
    at file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:81:23
    at read (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:78:15)
    at Module.readString (file:///usr/app/node_modules/@libp2p/multistream-select/src/multistream.ts:97:15)
    at Module.select (file:///usr/app/node_modules/@libp2p/multistream-select/src/select.ts:76:18)
Error: REQUEST_ERROR_DIAL_ERROR
    at sendRequest (file:///usr/app/packages/reqresp/src/request/index.ts:219:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ReqRespBeaconNode.sendRequest (file:///usr/app/packages/reqresp/src/ReqResp.ts:152:7)
    at collectExactOne (file:///usr/app/packages/reqresp/src/utils/collectExactOne.ts:11:20)
    at ReqRespBeaconNode.goodbye (file:///usr/app/packages/beacon-node/src/network/reqresp/ReqRespBeaconNode.ts:144:5)
    at PeerManager.goodbyeAndDisconnect (file:///usr/app/packages/beacon-node/src/network/peers/peerManager.ts:623:7)
Feb-10 20:00:40.896[network]       [36mverbose[39m: peer disconnected peer=16...hm5MfA, direction=outbound, status=CLOSING
Feb-10 20:00:41.005[network]         [34mdebug[39m: Req  sending request method=metadata, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1734, body=undefined
Feb-10 20:00:41.005[network]         [34mdebug[39m: Req  request sent method=metadata, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1734
Feb-10 20:00:41.309[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23692576, toBlock=23693575
Feb-10 20:00:41.334[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23693576, toBlock=23694575
Feb-10 20:00:41.925[network]       [36mverbose[39m: peer connected peer=16...cJj6Nw, direction=outbound, status=OPEN
Feb-10 20:00:41.925[network]         [34mdebug[39m: Req  dialing peer method=ping, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1739
Feb-10 20:00:41.925[network]         [34mdebug[39m: Req  dialing peer method=status, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1740
Feb-10 20:00:41.926[network]         [34mdebug[39m: Dialed discovered peer peer=16...cJj6Nw
Feb-10 20:00:41.926[network]         [34mdebug[39m: Dialed discovered peer peer=16...cJj6Nw
Feb-10 20:00:41.988[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23693576, toBlock=23694575
Feb-10 20:00:42.023[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23694576, toBlock=23695575
Feb-10 20:00:42.490[network]         [34mdebug[39m: Req  sending request method=ping, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1739, body=0
Feb-10 20:00:42.491[network]         [34mdebug[39m: Req  sending request method=status, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1740, body=undefined
Feb-10 20:00:42.494[network]         [34mdebug[39m: Resp received request method=goodbye, client=unknown, peer=16...cJj6Nw, requestId=1741, body=129
Feb-10 20:00:42.495[network]       [36mverbose[39m: Resp done method=goodbye, client=unknown, peer=16...cJj6Nw, requestId=1741
Feb-10 20:00:42.496[network]         [34mdebug[39m: Req  request sent method=ping, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1739
Feb-10 20:00:42.496[network]         [34mdebug[39m: Req  request sent method=status, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1740
Feb-10 20:00:42.496[network]       [36mverbose[39m: Received goodbye request peer=16...cJj6Nw, goodbye=129, reason=Client has too many peers
Feb-10 20:00:42.497[network]       [36mverbose[39m: Resp done method=goodbye, client=unknown, peer=16...cJj6Nw, requestId=1742
Feb-10 20:00:42.497[network]       [36mverbose[39m: Req  done method=ping, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1739
Feb-10 20:00:42.497[network]       [36mverbose[39m: Req  done method=status, encoding=ssz_snappy, client=undefined, peer=16...cJj6Nw, requestId=1740
Feb-10 20:00:42.498[network]       [36mverbose[39m: peer disconnected peer=16...cJj6Nw, direction=outbound, status=CLOSING
Feb-10 20:00:42.500[]                 [33mwarn[39m: Low peer count peers=1
Feb-10 20:00:42.500[]                 [32minfo[39m: Synced - slot: 7412460 (skipped 1) - head: 7412459 0x7b7f…34d5 - execution: valid(0x3983…dad1) - finalized: 0x4d6f…a462:463276 - peers: 1
Feb-10 20:00:42.706[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23694576, toBlock=23695575
Feb-10 20:00:42.731[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23695576, toBlock=23696575
Feb-10 20:00:43.352[chain]         [36mverbose[39m: Running prepareForNextSlot nextEpoch=463279, prepareSlot=7412461, headSlot=7412459, headRoot=0x7b7f6d83abc57d259f0f13b05be99da19ec5740e5197bdbf20aa3a137f8d34d5, isEpochTransition=false
Feb-10 20:00:43.405[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23695576, toBlock=23696575
Feb-10 20:00:43.433[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23696576, toBlock=23697575
Feb-10 20:00:44.080[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23696576, toBlock=23697575
Feb-10 20:00:44.106[eth1]          [36mverbose[39m: Fetched deposits depositCount=1, fromBlock=23697576, toBlock=23698575
Feb-10 20:00:44.748[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23697576, toBlock=23698575
Feb-10 20:00:44.767[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23698576, toBlock=23699575
Feb-10 20:00:45.000[chain]         [36mverbose[39m: Clock slot slot=7412461
Feb-10 20:00:45.360[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23698576, toBlock=23699575
Feb-10 20:00:45.380[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23699576, toBlock=23700575
Feb-10 20:00:45.890[network]       [36mverbose[39m: Received gossip block slot=7412461, root=0xed40…18b5, curentSlot=7412461, peerId=16Uiu2HAmDWN6iTXt7qGpJvcfppL8gHctSxtPB1eKp6rVvtwQSREX, delaySec=0.877000093460083, recvToVal=0.013000011444091797
Feb-10 20:00:45.892[network]         [34mdebug[39m: Gossip block has error slot=7412461, root=0xed40…18b5, code=BLOCK_ERROR_PARENT_UNKNOWN
Feb-10 20:00:45.894[network]         [34mdebug[39m: Req  dialing peer method=beacon_blocks_by_root, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1743
Feb-10 20:00:46.002[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23699576, toBlock=23700575
Feb-10 20:00:46.025[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23700576, toBlock=23701575
Feb-10 20:00:46.075[network]         [34mdebug[39m: Req  sending request method=beacon_blocks_by_root, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1743, body=0x768ee0c4059a39ac35adc0d8747cb4920c8ea51487b45fd241bb75c8ed56d6b4
Feb-10 20:00:46.076[network]         [34mdebug[39m: Req  request sent method=beacon_blocks_by_root, encoding=ssz_snappy, client=undefined, peer=16...wQSREX, requestId=1743
Feb-10 20:00:46.636[chain]           [34mdebug[39m: Persisted block to hot DB slot=7412460, root=0x768ee0c4059a39ac35adc0d8747cb4920c8ea51487b45fd241bb75c8ed56d6b4
Feb-10 20:00:46.637[chain]         [36mverbose[39m: Added block to forkchoice slot=7412460, root=0x768ee0c4059a39ac35adc0d8747cb4920c8ea51487b45fd241bb75c8ed56d6b4
Feb-10 20:00:46.664[chain]         [36mverbose[39m: New chain head slot=7412460, root=0x768ee0c4059a39ac35adc0d8747cb4920c8ea51487b45fd241bb75c8ed56d6b4, delaySec=6.664000034332275
Feb-10 20:00:46.665[chain]         [36mverbose[39m: Block processed slot=7412460, root=0x768ee0c4059a39ac35adc0d8747cb4920c8ea51487b45fd241bb75c8ed56d6b4, delaySec=6.664999961853027
Feb-10 20:00:46.787[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23700576, toBlock=23701575
Feb-10 20:00:46.808[eth1]          [36mverbose[39m: Fetched deposits depositCount=4, fromBlock=23701576, toBlock=23702575
Feb-10 20:00:46.904[chain]           [34mdebug[39m: Persisted block to hot DB slot=7412461, root=0xed40790d6f81037c663e3543d8f5cdfc1d91678dd78eb3312dba5deb922c18b5
Feb-10 20:00:46.905[chain]         [36mverbose[39m: Added block to forkchoice slot=7412461, root=0xed40790d6f81037c663e3543d8f5cdfc1d91678dd78eb3312dba5deb922c18b5
Feb-10 20:00:46.939[chain]         [36mverbose[39m: New chain head slot=7412461, root=0xed40790d6f81037c663e3543d8f5cdfc1d91678dd78eb3312dba5deb922c18b5, delaySec=1.938999891281128
Feb-10 20:00:46.940[chain]         [36mverbose[39m: Block processed slot=7412461, root=0xed40790d6f81037c663e3543d8f5cdfc1d91678dd78eb3312dba5deb922c18b5, delaySec=1.940000057220459
Feb-10 20:00:47.500[]                 [32minfo[39m: Synced - slot: 7412461 - head: 7412461 0xed40…18b5 - execution: valid(0xdb30…b848) - finalized: 0x4d6f…a462:463276 - peers: 1
Feb-10 20:00:47.542[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23701576, toBlock=23702575
Feb-10 20:00:47.560[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23702576, toBlock=23703575
Feb-10 20:00:48.156[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23702576, toBlock=23703575
Feb-10 20:00:48.176[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23703576, toBlock=23704575
Feb-10 20:00:48.350[chain]         [36mverbose[39m: Running prepareForNextSlot nextEpoch=463279, prepareSlot=7412462, headSlot=7412461, headRoot=0xed40790d6f81037c663e3543d8f5cdfc1d91678dd78eb3312dba5deb922c18b5, isEpochTransition=false
Feb-10 20:00:48.779[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23703576, toBlock=23704575
Feb-10 20:00:48.797[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23704576, toBlock=23705575
Feb-10 20:00:49.449[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23704576, toBlock=23705575
Feb-10 20:00:49.469[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23705576, toBlock=23706575
Feb-10 20:00:50.000[chain]         [36mverbose[39m: Clock slot slot=7412462
Feb-10 20:00:50.224[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23705576, toBlock=23706575
Feb-10 20:00:50.259[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23706576, toBlock=23707575
Feb-10 20:00:50.516[network]       [36mverbose[39m: Received gossip block slot=7412462, root=0x259f…227b, curentSlot=7412462, peerId=16Uiu2HAmDWN6iTXt7qGpJvcfppL8gHctSxtPB1eKp6rVvtwQSREX, delaySec=0.505000114440918, recvToVal=0.010999917984008789
Feb-10 20:00:50.567[chain]         [36mverbose[39m: Transitioned gossip block slot=7412462, recvToTransition=0.06199979782104492
Feb-10 20:00:50.590[chain]         [36mverbose[39m: Verified block signatures slot=7412462, recvToSigVer=0.08499979972839355
Feb-10 20:00:50.619[chain]         [36mverbose[39m: Verified execution payload slot=7412462, recvToVerifiedExecPayload=0.11399984359741211
Feb-10 20:00:50.848[chain]           [34mdebug[39m: Persisted block to hot DB slot=7412462, root=0x259ff6b2cd0c7a10a383f5de366c654b379b6d68b199d9d714251aa92fa6227b
Feb-10 20:00:50.849[chain]         [36mverbose[39m: Added block to forkchoice slot=7412462, root=0x259ff6b2cd0c7a10a383f5de366c654b379b6d68b199d9d714251aa92fa6227b
Feb-10 20:00:50.886[chain]         [36mverbose[39m: New chain head slot=7412462, root=0x259ff6b2cd0c7a10a383f5de366c654b379b6d68b199d9d714251aa92fa6227b, delaySec=0.8859999179840088
Feb-10 20:00:50.887[chain]         [36mverbose[39m: Imported block slot=7412462, recvToImportedBlock=0.3819999694824219
Feb-10 20:00:50.887[chain]         [36mverbose[39m: Block processed slot=7412462, root=0x259ff6b2cd0c7a10a383f5de366c654b379b6d68b199d9d714251aa92fa6227b, delaySec=0.8870000839233398
Feb-10 20:00:51.136[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23706576, toBlock=23707575
Feb-10 20:00:51.171[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23707576, toBlock=23708575
Feb-10 20:00:51.974[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23707576, toBlock=23708575
Feb-10 20:00:51.999[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23708576, toBlock=23709575
Feb-10 20:00:52.500[]                 [32minfo[39m: Synced - slot: 7412462 - head: 7412462 0x259f…227b - execution: valid(0x76fe…b65f) - finalized: 0x4d6f…a462:463276 - peers: 1
Feb-10 20:00:52.700[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23708576, toBlock=23709575
Feb-10 20:00:52.948[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23709576, toBlock=23710575
Feb-10 20:00:53.348[chain]         [36mverbose[39m: Running prepareForNextSlot nextEpoch=463279, prepareSlot=7412463, headSlot=7412462, headRoot=0x259ff6b2cd0c7a10a383f5de366c654b379b6d68b199d9d714251aa92fa6227b, isEpochTransition=false
Feb-10 20:00:53.670[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23709576, toBlock=23710575
Feb-10 20:00:53.696[eth1]          [36mverbose[39m: Fetched deposits depositCount=4, fromBlock=23710576, toBlock=23711575
Feb-10 20:00:54.370[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23710576, toBlock=23711575
Feb-10 20:00:54.401[eth1]          [36mverbose[39m: Fetched deposits depositCount=1, fromBlock=23711576, toBlock=23712575
Feb-10 20:00:55.000[chain]         [36mverbose[39m: Clock slot slot=7412463
Feb-10 20:00:55.117[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23711576, toBlock=23712575
Feb-10 20:00:55.139[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23712576, toBlock=23713575
Feb-10 20:00:55.757[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23712576, toBlock=23713575
Feb-10 20:00:55.775[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23713576, toBlock=23714575
Feb-10 20:00:56.171[network]       [36mverbose[39m: Received gossip block slot=7412463, root=0xafe1…0a4a, curentSlot=7412463, peerId=16Uiu2HAmDWN6iTXt7qGpJvcfppL8gHctSxtPB1eKp6rVvtwQSREX, delaySec=1.1619999408721924, recvToVal=0.00800013542175293
Feb-10 20:00:56.219[chain]         [36mverbose[39m: Transitioned gossip block slot=7412463, recvToTransition=0.057000160217285156
Feb-10 20:00:56.240[chain]         [36mverbose[39m: Verified block signatures slot=7412463, recvToSigVer=0.07800006866455078
Feb-10 20:00:56.259[chain]         [36mverbose[39m: Verified execution payload slot=7412463, recvToVerifiedExecPayload=0.0970001220703125
Feb-10 20:00:56.410[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23713576, toBlock=23714575
Feb-10 20:00:56.423[chain]           [34mdebug[39m: Persisted block to hot DB slot=7412463, root=0xafe127a6fe53978596357fc45c1ae1685b9421d502fe5178a42566269c390a4a
Feb-10 20:00:56.423[chain]         [36mverbose[39m: Added block to forkchoice slot=7412463, root=0xafe127a6fe53978596357fc45c1ae1685b9421d502fe5178a42566269c390a4a
Feb-10 20:00:56.443[chain]         [36mverbose[39m: New chain head slot=7412463, root=0xafe127a6fe53978596357fc45c1ae1685b9421d502fe5178a42566269c390a4a, delaySec=1.443000078201294
Feb-10 20:00:56.444[chain]         [36mverbose[39m: Imported block slot=7412463, recvToImportedBlock=0.2820000648498535
Feb-10 20:00:56.444[chain]         [36mverbose[39m: Block processed slot=7412463, root=0xafe127a6fe53978596357fc45c1ae1685b9421d502fe5178a42566269c390a4a, delaySec=1.444000005722046
Feb-10 20:00:56.451[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23714576, toBlock=23715575
Feb-10 20:00:57.193[eth1]          [36mverbose[39m: Fetched eth1 blocks blockCount=1000, fromBlock=23714576, toBlock=23715575
Feb-10 20:00:57.215[eth1]          [36mverbose[39m: Fetched deposits depositCount=0, fromBlock=23715576, toBlock=23716575
Feb-10 20:00:57.500[]                 [32minfo[39m: Synced - slot: 7412463 - head: 7412463 0xafe1…0a4a - execution: valid(0x321e…0da7) - finalized: 0x4d6f…a462:463276 - peers: 1

wemeetagain commented 1 year ago

Found the bug. For now you will need to explicitly set --network.maxPeers to some number eg: 55.

ThomasDalla commented 1 year ago

Nice. I can confirm the peers count is stable since adding that flag. Thanks!

ChainSafe / lodestar

Bug: Connected peers on Gnosis chain drops to 0 every 5 minutes #4816