Open ThomasBlock opened 7 months ago
update on this: threee of the systems work fine. one setup with etdocker still makes problems: ssv is rebooting altough execution and consensus client are totally fine.
ssv-node-1 | {"level":"info","time":"2024-03-26T13:17:45.292192Z","name":"execution_client","msg":"fetched registry events","from_block":19518848,"to_block":19518848,"target_block":19518848,"progress":"100.00%","events":0,"took":"5.556743ms"}
ssv-node-1 | {"level":"warn","time":"2024-03-26T13:17:45.601071Z","name":"Controller","msg":"failed to update validators metadata","error":"failed to get validator data from Beacon: failed to get validators data from beacon: failed to obtain validators: failed to obtain chunk: failed to request validators: failed to call GET endpoint: Get \"http://consensus:5052/eth/v1/beacon/states/head/validators?id=0x83d179a1f091fb06
....
context deadline exceeded\nfailed to get validators data from beacon\ngithub.com/bloxapp/ssv/protocol/v2/blockchain/beacon.FetchValidatorsMetadata\n\t/go/src/github.com/bloxapp/ssv/protocol/v2/blockchain/beacon/validator_metadata.go:113\ngithub.com/bloxapp/ssv/protocol/v2/blockchain/beacon.UpdateValidatorsMetadata\n\t/go/src/github.com/bloxapp/ssv/protocol/v2/blockchain/beacon/validator_metadata.go:71\ngithub.com/bloxapp/ssv/operator/validator.(*controller).UpdateValidatorMetaDataLoop\n\t/go/src/github.com/bloxapp/ssv/operator/validator/controller.go:858\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nfailed to get validator data from Beacon\ngithub.com/bloxapp/ssv/protocol/v2/blockchain/beacon.UpdateValidatorsMetadata\n\t/go/src/github.com/bloxapp/ssv/protocol/v2/blockchain/beacon/validator_metadata.go:73\ngithub.com/bloxapp/ssv/operator/validator.(*controller).UpdateValidatorMetaDataLoop\n\t/go/src/github.com/bloxapp/ssv/operator/validator/controller.go:858\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}
ssv-node-1 | {"level":"info","time":"2024-03-26T13:18:07.316673Z","name":"P2PNetwork","msg":"Verified handshake nodeinfo","selfPeer":"16Uiu2HAmVd4pPhEMR5RnAboqftLzuqZHiZy1CzLpdRP9qbFsoWxh","peer_id":"16Uiu2HAm6bqkkkkKpnHqzgrxjmJ57mNCe9Ph4MN7LdhkPedKG77h","peer_id":"16Uiu2HAm6bqkkkkKpnHqzgrxjmJ57mNCe9Ph4MN7LdhkPedKG77h","metadata":{"NodeVersion":"v1.3.2-97d20e67d83cad1fd0d8d12ff179f7a9fe090daa","ExecutionNode":"","ConsensusNode":"","Subnets":"f5ffffffffbe3ebbdbf7fffbff766c6b"},"networkID":"0x00000000"}
ssv-node-1 | {"level":"info","time":"2024-03-26T13:18:07.852493Z","name":"execution_client","msg":"fetched registry events","from_block":19518849,"to_block":19518849,"target_block":19518849,"progress":"100.00%","events":0,"took":"660.656µs"}
ssv-node-1 | {"level":"error","time":"2024-03-26T13:18:07.874067Z","msg":"node is not healthy","node":"consensus client","error":"failed to obtain node syncing status: failed to call GET endpoint: Get \"http://consensus:5052/eth/v1/node/syncing\": context deadline exceeded"}
ssv-node-1 | {"level":"error","time":"2024-03-26T13:18:07.874151Z","msg":"not all nodes are healthy"}
ssv-node-1 | {"level":"fatal","time":"2024-03-26T13:18:07.874164Z","msg":"ethereum node(s) are either out of sync or down. Ensure the nodes are healthy to resume."}
This is Eth Docker v2.8.0.0
ssvnode version v1.3.2-97d20e67d83cad1fd0d8d12ff179f7a9fe090daa
beacon-chain version Prysm/v5.0.1/a1a81d1720a0a3b850992d4825d0a023baa8e65a. Built at: 2024-03-08 20:21:37+00:00
validator version Prysm/v5.0.1/a1a81d1720a0a3b850992d4825d0a023baa8e65a. Built at: 2024-03-08 20:22:56+00:00
besu/v24.3.0/linux-x86_64/openjdk-java-17
mev-boost v1.7.1
update: was okay for a long time. 6 of 7 nodes work fine. now problems with this setup. 10 reboots of ssv-node a day bring performance down to 86 % - all while consensus and execution client are fine..
Yellow = good node Blue= bad node
ethd version
This is Eth Docker v2.9.2.0
ssvnode version v1.3.4-39046e4aa45ab4b2d8bd48af41d62bc5858c59ad
beacon-chain version Prysm/v5.0.3/38f208d70dc95b12c08403f5c72009aaa10dfe2f. Built at: 2024-04-04 18:29:14+00:00
2024-06-16 15-09-07.7946|Nethermind starting initialization.
2024-06-16 15-09-07.8395|Client version: Nethermind/v1.26.0+0068729c/linux-x64/dotnet8.0.4
Describe the bug I am experimenting with different SSV setups. most are working fine. But this configuration crashes several times a day. It reboots, but would be nice to avoid these completely..
To Reproduce ubuntu22 ethdocker geth nimbus-cl-only SSV-Node:v1.2.3
Logs