Lighthouse/Lighthouse combo (running with distributed flag), although gets slightly better over time
, is experiencing issues with missing duties, specifically:
aggregate_attestation duties are being missed.
sync_committee_contribution duties are being missed.
This could be due to another misconfiguration probably in kurtosis. Also, the beacon node score is low (97%), and VC loads keys extremely slow (15 minutes or longer).
VC log:
2024-05-17 11:41:02 May 17 08:41:02.002 INFO All validators active slot: 120, epoch: 3, total_validators: 600, active_validators: 600, current_epoch_proposers: 11, service: notifier
2024-05-17 11:41:04 May 17 08:41:04.002 DEBG Request to beacon node failed error: HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out), node: http://node0:3600/
2024-05-17 11:41:04 May 17 08:41:04.002 CRIT Failed to produce sync contribution error: Some endpoints failed, num_failed: 1 http://node0:3600/ => RequestFailed(HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out)), beacon_block_root: 0xf39e22a862b1b632a791c716fed6b5196b6dfac88c41ed69185b189f9fdc3a1c, slot: 119, service: sync_committee
2024-05-17 11:41:04 May 17 08:41:04.002 DEBG Request to beacon node failed error: HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out), node: http://node0:3600/
2024-05-17 11:41:04 May 17 08:41:04.002 CRIT Failed to produce sync contribution error: Some endpoints failed, num_failed: 1 http://node0:3600/ => RequestFailed(HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out)), beacon_block_root: 0xf39e22a862b1b632a791c716fed6b5196b6dfac88c41ed69185b189f9fdc3a1c, slot: 119, service: sync_committee
2024-05-17 11:41:04 May 17 08:41:04.002 DEBG Request to beacon node failed error: HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out), node: http://node0:3600/
2024-05-17 11:41:04 May 17 08:41:04.002 CRIT Failed to produce sync contribution error: Some endpoints failed, num_failed: 1 http://node0:3600/ => RequestFailed(HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out)), beacon_block_root: 0xf39e22a862b1b632a791c716fed6b5196b6dfac88c41ed69185b189f9fdc3a1c, slot: 119, service: sync_committee
2024-05-17 11:41:04 May 17 08:41:04.002 DEBG Request to beacon node failed error: "Failed to produce an aggregate attestation: HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out)", node: http://node0:3600/
2024-05-17 11:41:04 May 17 08:41:04.002 CRIT Error during attestation routine slot: 119, committee_index: 0, error: "Some endpoints failed, num_failed: 1 http://node0:3600/ => RequestFailed(\"Failed to produce an aggregate attestation: HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out)\")", service: attestation
2024-05-17 11:41:04 May 17 08:41:04.002 DEBG Request to beacon node failed error: HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out), node: http://node0:3600/
2024-05-17 11:41:04 May 17 08:41:04.002 CRIT Failed to produce sync contribution error: Some endpoints failed, num_failed: 1 http://node0:3600/ => RequestFailed(HttpClient(url: http://node0:3600/, kind: timeout, detail: operation timed out)), beacon_block_root: 0xf39e22a862b1b632a791c716fed6b5196b6dfac88c41ed69185b189f9fdc3a1c, slot: 119, service: sync_committee
🎯 Problem to be solved
Lighthouse/Lighthouse combo (running with
distributed
flag), although gets slightly better over time , is experiencing issues with missing duties, specifically:aggregate_attestation
duties are being missed.sync_committee_contribution
duties are being missed.This could be due to another misconfiguration probably in kurtosis. Also, the beacon node score is low (97%), and VC loads keys extremely slow (15 minutes or longer).
VC log:
charon log:
CL error:
🛠️ Proposed solution
aggregate_attestation
duties and address any underlying issues.sync_committee_contribution
duties and address any underlying issues.validators.json
file, which may help improve the key loading performance.run.sh
script.