Metrics to add tracker

[dapplion]

Ready to execute

• https://github.com/ChainSafe/lodestar/issues/2483
• https://github.com/ChainSafe/lodestar/issues/2984
• https://github.com/ChainSafe/lodestar/issues/3033
• https://github.com/ChainSafe/lodestar/issues/3112
• https://github.com/ChainSafe/lodestar/issues/3115
• https://github.com/ChainSafe/lodestar/issues/3205
• https://github.com/ChainSafe/lodestar/issues/3413
• https://github.com/ChainSafe/lodestar/issues/3680
• https://github.com/ChainSafe/lodestar/issues/3946
• https://github.com/ChainSafe/lodestar/pull/4122 ReqResp request times
• https://github.com/ChainSafe/lodestar/pull/4123 Reorg depth histogram
• https://github.com/ChainSafe/lodestar/pull/4123 Distance to parent block histogram
• https://github.com/ChainSafe/lodestar/pull/4124 Histogram of balance increase for block proposer
• https://github.com/ChainSafe/lodestar/pull/4125 Total signatures aggregated (optional histogram for time spent aggregating)
• https://github.com/ChainSafe/lodestar/pull/4127 JSON RPC client metrics
• From Lighthouse: https://github.com/ChainSafe/lodestar/issues/4273
• From Lighthouse: https://github.com/ChainSafe/lodestar/issues/4274
• From Lighthouse: https://github.com/ChainSafe/lodestar/issues/2558
• From Lighthouse: https://github.com/ChainSafe/lodestar/issues/2984
• From Lighthouse: https://github.com/ChainSafe/lodestar/issues/2473

Metric ideas

Misc

Once properly defined, convert to issue or pull request and move above to "Ready to execute"

• Improve state.clone() metrics, what we really care about is before doing state-transitions if the state has cache or not.
• Add more sync metrics to measure efficiency: # blocks downloaded to process # of blocks. To count duplicated downloads indirectly.
• Discv5 ping times
• Gossipsub received blocks
• Blocks in last epoch, to compute percent of orphaned
• More HTTP metrics?

From Lighthouse

Reviewed this file and select interesting metrics to add https://github.com/sigp/lighthouse/blob/stable/beacon_node/beacon_chain/src/metrics.rs

Block Delay Metrics

• [x] beacon_block_imported_slot_start_delay_time: Duration between the start of the blocks slot and the current time when it was imported.
• [x] beacon_block_head_slot_start_delay_time: Duration between the start of the blocks slot and the current time when it was as head.
• [ ] beacon_block_head_slot_start_delay_exceeded_total: Triggered when the duration between the start of the blocks slot and the current time will result in failed attestations. https://github.com/ChainSafe/lodestar/pull/5559

Persisting BeaconChain components to disk

• [ ] beacon_persist_head: Time taken to persist the canonical head
• [ ] beacon_persist_op_pool: Time taken to persist the operations pool - Does Lighthouse keep all ops in memory?
• [ ] beacon_persist_eth1_cache: Time taken to persist the eth1 caches

Chain Head

• [x] beacon_update_head_seconds: Time taken to update the canonical head - Review what does it refer too

[g11tech]

assign please

[g11tech]

@dapplion these looks like good metrics to pick up next ?

[dapplion]

@dapplion these looks like good metrics to pick up next ?

I think the next interesting group to add would be around block production. Note that lighthouse has very detailed timings on each step of the process. Our approach to block production is more simple so maybe a single histogram around the entire thing would suffice. We are already kind of tracking it via the API response times, but it would be nice to track it anyway.

• beacon_block_production_requests_total: Count of all block production requests
• beacon_block_production_successes_total: Count of blocks successfully produced
• beacon_block_production_seconds: Full runtime of block production

At the same time it would be quite a priority to have metrics in our regen / state cache: https://github.com/ChainSafe/lodestar/issues/2153. Let's discuss details in that issue if you are interested.

[dapplion]

Keeping open since there are some metrics we want to implement soonish too

[philknows]

Adding some ideas to this from @wemeetagain discussion in Columbia that I found in my notes:

• Block size metrics (due to 4844 blobs) to test performance
• Number of operations per block
• Metrics on exits, slashings, deposits.