Huge network thread latency to/from main thread when subscribing to all subnets

[twoeths]

Describe the bug

When subscribing to all subnets, network thread to/from main thread latency increase significantly

I don't see any latency from libp2p, or at least it's a tiny difference, we still receive gossip block on time at network thread

the above latency leads to significantly increase of below topics:

• Avg block recv to gossip validation delay

• Avg block recv to state transition delay

Expected behavior

for beacon_block topic, we don't want to have this latency, as it causes block to becomes head very late and validator votes for wrong head, see #7186

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.22.0, unstable

[twoeths]

I had a hypothesis that queued attestations at the main thread side may cause this, so I queued at network thread instead but it did not work, see #7189

it seems to me the gc caused this (before 23:00 is the version without subscribe_to_subnet flag)

[twoeths]

the validator performance really depends on latency of beacon block, this metric track the time when we first seen beacon block in network thread until the time it's validated in main thread

it really looks like gc caused beacon blocks to be delayed frequently when it runs due to that validator votes for wrong block head/target and it may cause missed attestations for those blocks