paulhauner
commented
4 years ago I have discussed this offline with @agemanning, but I'll leave this here for reference :)
The metrics show attestations after verification, so this could be due to invalid attestations or non-participation.
We have metrics prior to verification and they're showing roughly 300-1000 aggregate attestations per minute coming in from gossipsub into the BeaconProcessor.
Which is significantly less than expected (even accounting for the current participation rate)
I think that the numbers we're seeing are reasonable. Firstly, I think we can expect at most 560 aggregators per minute:
>>> MEDALLA_VALS = 30000
>>> SLOTS_PER_EPOCH = 32
>>> TARGET_COMMITTEE_SIZE = 128
>>> COMMITTEE_COUNT_PER_SLOT = MEDALLA_VALS // SLOTS_PER_EPOCH // TARGET_COMMITTEE_SIZE
>>> TARGET_AGGREGATORS_PER_COMMITTEE = 16
>>> SLOTS_PER_MINUTE = 60 // 12
>>> COMMITTEE_COUNT_PER_SLOT * TARGET_AGGREGATORS_PER_COMMITTEE * SLOTS_PER_MINUTE
560Furthermore, we can lower this expected number for two reasons:
- There a bunch of validators (~30%) offline presently on Medalla (as mentioned).
- The networking spec declares that we should "IGNORE" any attestations which contain an
aggregatethat has already been seen. If the network is working perfectly, or even well, then a majority of the aggregated attestation should be the same and we should observe them being suppressed on gossip.
In fact, it seems like we're seeing a few too many aggregate attestations flying around.
alex3498
adetante
AgeManning
On the chart, you can see that CPU load is around 20% but it is actually fully loaded in 
jeffla
Description
Metrics are reporting seeing significantly less attestations on the network than we should see theoretically. The metrics show attestations after verification, so this could be due to invalid attestations or non-participation. However we are an order of magnitude out and it would be nice to get an handle on why so many are missing.
The easiest place to start, is to look at the aggregated attestations. For each subnet, we expect to have (on average) 16 aggregators per subnet and there should be 64 subnets per slot. We therefore should be seeing 1024 aggregated attestations per slot.
Currently we are witnessing about (for all attestations, not just aggregates):
Which is significantly less than expected (even accounting for the current participation rate). It might be useful to split the metrics into unaggregated attestations and aggregated attestations in the metrics to get a clearer distinction. Trying to isolate the main causes for this discrepency will help improve our attestation inclusion rate. See here: https://github.com/sigp/lighthouse/blob/master/beacon_node/network/src/beacon_processor/mod.rs#L952 For where the metrics are logged for all attestations.