SimonSMH1015
commented
10 months ago Got a similar problem. The RAM usage of my lighthouse keeps increasing and then falling back to normal sharply.
Are this related to issue: https://github.com/sigp/lighthouse/issues/4918 ?
During this period, the following error information is recorded in the log: (I've only cited a few logs. Info-level logs are no different. I'm happy to provide complete logs if necessary.)
Nov 26 09:26:40.424 ERRO Ignored scheduled attestation(s) for block, sent_count: 12288, failed_count: 2875, block_root: 0x8f4a6f2d72c5f455556966f5f08f81a5104a0f994d635de1a846a6853530a3f6, parent_root: 0xdad5dd2962ef73e80bf76f487961e781d8d1d0530ffc5faab45edbe5bc8c7bc2, hint: system may be overloaded, service: bproc
Nov 26 09:03:13.562 ERRO Attestation delay queue is full msg: check system clock, queue_size: 16384, service: bproc
Nov 26 08:50:54.771 ERRO Ignored scheduled attestation(s) for block, sent_count: 12288, failed_count: 4096, block_root: 0x212a77c83debe9dccbc10f564d6d0e959342e81779dcd8aac3a08a73d5ad2cac, parent_root: 0x0502f524a50b9fbeef0a9ae29eef952347610f7472a82942465eb2d52026eff6, hint: system may be overloaded, service: bproc
Nov 26 08:50:52.106 ERRO Attestation delay queue is full msg: check system clock, queue_size: 16384, service: bproc
Nov 26 08:20:52.164 ERRO Ignored scheduled attestation(s) for block, sent_count: 12288, failed_count: 4096, block_root: 0x391f477c7022235b38cf6ff315984ed81b04e783e6f40bdb71491818b3d62aa3, parent_root: 0x29355ddd821bd581e466f4f4436b48933eab66bc851b6c451108c795cbe2be54, hint: system may be overloaded, service: bproc
Nov 26 08:20:51.779 ERRO Attestation delay queue is full msg: check system clock, queue_size: 16384, service: bproc
michaelsproul
AgeManning
tobidae-cb
adaszko
Description
Our lighthouse container running on Kubernetes keeps having intermittent bursts of the Attestation Queue Full error. I initially tried bumping the compute to use 40vCPU and 200Gi of RAM on a i3en.24xlarge compute but that's already starting to fail.
This issue after a while clears up but as it continues to come up, it's affecting smooth operations of our Nodes. I'll provide additional details like logs, e.t.c below.
The lighthouse configuration is meant to act as an archival consensus deployment so we have
--slots-per-restore-point=32set as a beacon node flagVersion
441fc1691b69f9edc4bbdc6665f3efab16265c9bPresent Behaviour
At a random time, our node will be syncing, then stop displaying the
New block receivedmessage. After roughly 2 minutes, theAggregate attestation queue fullmessage starts displaying.About 3 minutes before this happens, there is a spike in the amount of CPU being used from 3vCPUs to 55 before settling at 40vCPU. The RAM usage also spikes from 7GiB to 50GiB (went to 84GiB but settled at 50).
While writing this message the issue self resolved, I'm guessing once the queue was cleared.
LOGS
Configuration
Compute Config
Expected Behaviour
Lighthouse should not be having this issue given that we're currently not making any API calls to the beacon endpoints, only geth is interacting with the lighthouse deployment.
Steps to resolve