search

status-im / nimbus-eth2

Nim implementation of the Ethereum Beacon Chain
https://nimbus.guide
Other
527 stars 229 forks source link

reload of w3s #6603

Open franjoespejo opened 1 day ago

franjoespejo commented 1 day ago

Describe the bug When the connection with the w3s is closed and the requests do timeout, if you enable the w3s, the connection become active again and the pubkeys reloaded start to detach and attach pubkeys from the w3s but after they do this the logs stop and validator freezes but doesnt crash

To Reproduce Steps to reproduce the behavior:

  1. Platform details (OS, architecture): 'amd64, ubuntu'
  2. Branch/commit used: ' version{version="v24.5.1-d2a075-stateofus",commit="d2a075"} 1.0 version{version="v24.7.0-99f657-stateofus",commit="99f657"} 1.0 version{version="v24.9.0-f54a03-stateofus",commit="f54a03"} 1.0 '
  3. Commands being executed: ' ENTRYPOINT: nimbus_validator_client CMD: --doppelganger-detection=false --non-interactive --web3-signer-update-interval=10 --beacon-node=http://172.16.0.13:4000 --suggested-fee-recipient=0x7d58F316a09064cA16c23Ad9c8e3Ab37E51815Ff --graffiti=val1 --metrics --metrics-address=0.0.0.0 --metrics-port=34050 --web3-signer-url=http://diva5:9000 '
  4. Relevant log lines:

'WRN 2024-10-06 16:20:31.003+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1136 validator=b43ccb05@67 block_root=632972d4 WRN 2024-10-06 16:20:31.003+00:00 Unable to sign attestation reason="Operation timed out" service=attestation_service validator=b97dbe4a@72 WRN 2024-10-06 16:20:31.004+00:00 Unable to sign attestation reason="Operation timed out" service=attestation_service validator=b97ecbcf@65 WRN 2024-10-06 16:20:31.166+00:00 Unable to resolve validator's source distributed signer address reason="Unable to resolve remote hostname" web3signer_url=http://diva1:9000 INF 2024-10-06 16:20:32.001+00:00 Slot start slot=1137 epoch=142 attestationIn=2s blockIn=12s validators=10 node_status=synced delay=1ms540us769ns WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1138 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1138 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1138 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139 WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1140 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1140 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1140 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1143 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1143 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1143 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1145 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1145 WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1145 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144 WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144 WRN 2024-10-06 16:20:34.140+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1137 validator=b43ccb05@67 block_root=d307bfd3 WRN 2024-10-06 16:20:34.140+00:00 Unable to sign attestation reason="Remote signer returns 500 Internal Server Error" service=attestation_service validator=ac30aacd@66 WRN 2024-10-06 16:20:34.140+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1137 validator=8de7ec50@64 block_root=d307bfd3 WRN 2024-10-06 16:20:34.140+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1137 validator=ac30aacd@66 block_root=d307bfd3 WRN 2024-10-06 16:20:34.142+00:00 Unable to sign attestation reason="Remote signer returns 500 Internal Server Error" service=attestation_service validator=a39731d5@71 WRN 2024-10-06 16:20:34.142+00:00 Unable to sign attestation reason="Remote signer returns 500 Internal Server Error" service=attestation_service validator=8de7ec50@64 NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=ac30aacd9e91cb0727c34ca6b40fbfd4d255b998471e25c443cf6cf777d6bb823a58e162958f32c3c5ca80453387a5d2 validator=ac30aacd NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=966c488d807b3208bb1b10a1af422bac8d363c8015cda4e24d214549ced019cd3dd575545dd887461cae3f70d95cb061 validator=966c488d NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=b8c6663371dc37bf083134dea26a20115ccc52b7c15a662bcfa33435e4ade14c6bc9714a5cdee492530accf8a327b2aa validator=b8c66633 NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=b3faeebfbebd085b9123ae0e09af9cd15d3b1db6a25f3e82d8b48b68e53522b41b342a3a3c8b008897df356048862d98 validator=b3faeebf NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=b97dbe4add8aefd96c575ae9de19d1ac590bb7d92f23a9e4e113f7271c2243cf689e7645879efbb546d58ec44f5263d6 validator=b97dbe4a NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=a39731d5cb52838d02d4ff897ab908c0f76a9ef837f9288c634ed3091a1f69d5347dc65cd2c8009a5207a369a4c6bdae validator=a39731d5 NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=8de7ec501d574152f52a962bf588573df2fc3563fd0c6077651208ed20f24f3d8572425706b343117b48bdca56808416 validator=8de7ec50 NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=b97ecbcfe8c52b9bcdca9e75da13c5650b751b037c570934ea6b6441ff32de6566c50dafc0557e63105b2ee7e8cbb39e validator=b97ecbcf NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=95833097520df43a5cb013e97f80041a7a0b7d84a4ec79e2f16baeeb6edfbcf62ede97becfde73883831bb65e1415dc0 validator=95833097 NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached topics="val_pool" pubkey=b43ccb05317c2b666470ab251e987d6bf31f5ead6b5edac5fe007dd334ae6ce1a92e24c19e5ab387cd8fa253b63bb78c validator=b43ccb05 INF 2024-10-06 16:20:38.002+00:00 Slot start slot=1138 epoch=142 attestationIn=8s blockIn= validators=0 node_status=synced delay=2ms728us102ns INF 2024-10-06 16:20:44.001+00:00 Slot start slot=1139 epoch=142 attestationIn=2s blockIn= validators=0 node_status=synced delay=1ms402us782ns NTC 2024-10-06 16:20:46.169+00:00 Dynamic remote validator attached topics="val_pool" pubkey=b8c6663371dc37bf083134dea26a20115ccc52b7c15a662bcfa33435e4ade14c6bc9714a5cdee492530accf8a327b2aa validator=b8c66633 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b8c6663371dc37bf083134dea26a20115ccc52b7c15a662bcfa33435e4ade14c6bc9714a5cdee492530accf8a327b2aa)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached topics="val_pool" pubkey=a765e534ca5f091ef4056dac0b84bcdbc90751bc0950b46cccebbe569f3f6e8afc964cf7d34adfed9b444e6d6d300d6d validator=a765e534 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: a765e534ca5f091ef4056dac0b84bcdbc90751bc0950b46cccebbe569f3f6e8afc964cf7d34adfed9b444e6d6d300d6d)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached topics="val_pool" pubkey=b97ecbcfe8c52b9bcdca9e75da13c5650b751b037c570934ea6b6441ff32de6566c50dafc0557e63105b2ee7e8cbb39e validator=b97ecbcf remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b97ecbcfe8c52b9bcdca9e75da13c5650b751b037c570934ea6b6441ff32de6566c50dafc0557e63105b2ee7e8cbb39e)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached topics="val_pool" pubkey=b97dbe4add8aefd96c575ae9de19d1ac590bb7d92f23a9e4e113f7271c2243cf689e7645879efbb546d58ec44f5263d6 validator=b97dbe4a remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b97dbe4add8aefd96c575ae9de19d1ac590bb7d92f23a9e4e113f7271c2243cf689e7645879efbb546d58ec44f5263d6)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached topics="val_pool" pubkey=966c488d807b3208bb1b10a1af422bac8d363c8015cda4e24d214549ced019cd3dd575545dd887461cae3f70d95cb061 validator=966c488d remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 966c488d807b3208bb1b10a1af422bac8d363c8015cda4e24d214549ced019cd3dd575545dd887461cae3f70d95cb061)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached topics="val_pool" pubkey=b3faeebfbebd085b9123ae0e09af9cd15d3b1db6a25f3e82d8b48b68e53522b41b342a3a3c8b008897df356048862d98 validator=b3faeebf remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b3faeebfbebd085b9123ae0e09af9cd15d3b1db6a25f3e82d8b48b68e53522b41b342a3a3c8b008897df356048862d98)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached topics="val_pool" pubkey=8de7ec501d574152f52a962bf588573df2fc3563fd0c6077651208ed20f24f3d8572425706b343117b48bdca56808416 validator=8de7ec50 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 8de7ec501d574152f52a962bf588573df2fc3563fd0c6077651208ed20f24f3d8572425706b343117b48bdca56808416)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached topics="val_pool" pubkey=87df22a26f59f2854d697ecfda8903f3ea6dd9ec91ae7effb5ff6ab0d599b7d3c57c1137d21522e1ab2adb8887e3a1c4 validator=87df22a2 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 87df22a26f59f2854d697ecfda8903f3ea6dd9ec91ae7effb5ff6ab0d599b7d3c57c1137d21522e1ab2adb8887e3a1c4)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached topics="val_pool" pubkey=ac30aacd9e91cb0727c34ca6b40fbfd4d255b998471e25c443cf6cf777d6bb823a58e162958f32c3c5ca80453387a5d2 validator=ac30aacd remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: ac30aacd9e91cb0727c34ca6b40fbfd4d255b998471e25c443cf6cf777d6bb823a58e162958f32c3c5ca80453387a5d2)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached topics="val_pool" pubkey=95833097520df43a5cb013e97f80041a7a0b7d84a4ec79e2f16baeeb6edfbcf62ede97becfde73883831bb65e1415dc0 validator=95833097 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 95833097520df43a5cb013e97f80041a7a0b7d84a4ec79e2f16baeeb6edfbcf62ede97becfde73883831bb65e1415dc0)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached topics="val_pool" pubkey=b43ccb05317c2b666470ab251e987d6bf31f5ead6b5edac5fe007dd334ae6ce1a92e24c19e5ab387cd8fa253b63bb78c validator=b43ccb05 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b43ccb05317c2b666470ab251e987d6bf31f5ead6b5edac5fe007dd334ae6ce1a92e24c19e5ab387cd8fa253b63bb78c)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000 NTC 2024-10-06 16:20:46.172+00:00 Dynamic remote validator attached topics="val_pool" pubkey=a39731d5cb52838d02d4ff897ab908c0f76a9ef837f9288c634ed3091a1f69d5347dc65cd2c8009a5207a369a4c6bdae validator=a39731d5 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: a39731d5cb52838d02d4ff897ab908c0f76a9ef837f9288c634ed3091a1f69d5347dc65cd2c8009a5207a369a4c6bdae)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000'


that's all the logs, the validator keeps running but doesnt work nor log
franjoespejo commented 1 day ago

it happens for both minimal and mainnet preset with different interval values from 10 to 360s

cheatfate commented 12 hours ago

It looks like web3signer client unable to resolve hostname of web3signer host which is diva1.