Delayed IWANT messages from Lodestar to Nimbus

Description Some how IWANT messages from Lodestar are not reached Nimbus on time and it causes broken promises: a lot of Nimbus peers has low gossipsub score. For example:

Lodestar log:

Feb-22 09:03:44.676[network]         ^[[34mdebug^[[39m: 2023-02-22T09:03:44.676Z libp2p:gossipsub IHAVE: Asking for %d out of %d messages from %s 2 2

grep -ie "5B3879745F5E9E600253114F0CD239ACDE7D3F9B" -rn beacon*.log
beacon-2023-02-22.log:34554647:Feb-22 09:03:44.676[network]         debug: 2023-02-22T09:03:44.676Z libp2p:gossipsub IHAVE: Asking from peers 16Uiu2HAmThKa44DYZKvYi52z5gKyAkdLydMrEewrYQHGn9Ez64uS message id 5b3879745f5e9e600253114f0cd239acde7d3f9b
beacon-2023-02-22.log:34590346:Feb-22 09:03:57.282[network]         debug: 2023-02-22T09:03:57.282Z libp2p:gossipsub peer %s didn't follow up in %s IWANT requests; adding penalty 16Uiu2HAmThKa44DYZKvYi52z5gKyAkdLydMrEewrYQHGn9Ez64uS 5b3879745f5e9e600253114f0cd239acde7d3f9b

Nimbus log

{"lvl":"INF","ts":"2023-02-22 09:04:01.128+00:00","msg":"LATE IWANT","topics":"libp2p gossipsub","diff":{"value":16664079985},"peerType":"lodestar","id":"2D73F92464D50A5EAE0CA579AB6BA3776D5B5D8E","peerId":"16U*jAc99o"}
{"lvl":"INF","ts":"2023-02-22 09:04:01.128+00:00","msg":"LATE IWANT","topics":"libp2p gossipsub","diff":{"value":17769000590},"peerType":"lodestar","id":"5B3879745F5E9E600253114F0CD239ACDE7D3F9B","peerId":"16U*jAc99o"}

===> it takes 15s to reach Nimbus. After 12s Lodestar marked that as a broken promise, Nimbus was not able to response respective messages since it's too late.

This issue does not always happen, for example below IWANT messages were sent to another Nimbus peer and it responded timely

grep -ie "2D73F92464D50A5EAE0CA579AB6BA3776D5B5D8E" -rn beacon*.log
beacon-2023-02-22.log:34557475:Feb-22 09:03:45.659[network]         debug: 2023-02-22T09:03:45.659Z libp2p:gossipsub IHAVE: Asking from peers 16Uiu2HAm9r2G4E49mKUVZgMXM63LwqjMNtb9g3dNsMKJAg1yCc8Y message id 2d73f92464d50a5eae0ca579ab6ba3776d5b5d8e
beacon-2023-02-22.log:34557648:Feb-22 09:03:45.802[network]         debug: 2023-02-22T09:03:45.802Z libp2p:gossipsub Received valid message 2d73f92464d50a5eae0ca579ab6ba3776d5b5d8e from 16Uiu2HAm9r2G4E49mKUVZgMXM63LwqjMNtb9g3dNsMKJAg1yCc8Y after 143ms

ChainSafe / js-libp2p-gossipsub

Delayed IWANT messages from Lodestar to Nimbus #408