wealdtech / chaind

Apache License 2.0
98 stars 32 forks source link

summarizer does not progress #84

Open kris-w-stkd opened 1 year ago

kris-w-stkd commented 1 year ago

I configured chaind to start at epoch 171176 and slot 5477649 (roughly last two months).

many of the tables have progressed to the chain head as expected, e.g. t_blocks and t_attestations, however the table t_validator_epoch_summaries remains empty

as I understood from reading the other issues, the summarizer will wait for the finalizer service to progress, from what I can see in the logs I think it is progressing, the most recent tail logs look like this:

{"level":"error","service":"finalizer","impl":"standard","finalized_epoch":183457,"error":"failed to obtain finality for state: failed to request finality checkpoints: failed to call GET endpoint: Get \"node_address_redacted/eth/v1/beacon/states/5869632/finality_checkpoints\": EOF","time":"2023-02-24T21:01:08Z","message":"Failed to build finality stack"}

The error message from the finalizer service is progressing forward, so I suspected that It fails and posts the error, but then succeeds on a later retry and progresses. It always seems to be a couple epochs behind the chain head that I see on beaconcha.in for example.

Is there a way to confirm the output of the finalizer service? And is there a way to get the validator epoch summaries to progress, e.g. by disabled the finalizer service so that the summarizer at least runs on the older blocks or something?

thanks

mcdee commented 1 year ago

You can check the state of the finalizer by finding out which blocks are yet to be marked as canonical or not. For example:

SELECT MIN(f_slot) FROM t_blocks WHERE f_canonical IS NULL

should be increasing as the epochs tick by and the finalizer does its job.

If the finalizer is increasing them the summarizer should also be running, as it is triggered from the finalizer. You can check the state of finalization with:

SELECT MAX(f_epoch) FROM t_epoch_summaries

Again, this should be increasing every time an epoch finalizes.