search

ChainSafe / lodestar

🌟 TypeScript Implementation of Ethereum Consensus
https://lodestar.chainsafe.io
Apache License 2.0
1.19k stars 291 forks source link

Peer cronically behind #3110

Closed dapplion closed 2 years ago

dapplion commented 3 years ago

Describe the bug

Our peer is cronically behind getting blocks very late from mesh peers.

In this logs you can see how three Lighthouse peers discover the same block 60s late and forward to us in sequence:

Scroll down starting at log Eph 63940/26 6.001 [] info: Synced - block: 2046106

                  >>> Gossipssub - block 2046100 (new)           46.27s late 6x6DsvCkteL7pg/cYRHZMyOcXH8 Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
Eph 63940/23 10.284 [CHAIN]            info: Received block: 2046100
                  >>> Gossipssub - block 2046092 (new)           142.69s late 8gBwsSrvZL+1Hg9cKMC0v/zzr8Y Prysm/v1.3.1/6e831920bfa08a300d2786f7039314b03ebbd3ab 16Uiu2HAmLeVegSHYfm9RAebtKetEEGmmebR1VJLBk7KXaYNVsVT6
                  >>> Gossipssub - block 2046100 (already known) 46.85s late 6x6DsvCkteL7pg/cYRHZMyOcXH8 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
Eph 63940/24 0.504 [SYNC]             info: INFURA HEAD 2046104, delay 0.5039999485015869s
                  >>> Gossipssub - block 2046099 (already known) 64.67s late ynTrcsYGjvICvPAlPg6ltVdDRDQ Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046099 (already known) 64.81s late ynTrcsYGjvICvPAlPg6ltVdDRDQ rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
                  >>> Gossipssub - block 2046094 (new)           124.97s late wGY/LTzfteifAFQdDbQuhUCZAk4 Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
Eph 63940/24 6.000 []                 info: Synced - block: 2046104   … (empty) - head: 2046100 0x519d…3338 - finalized: 2046016 0x6b11…cf1c (63938) - peers: 12
                  >>> Gossipssub - block 2046092 (already known) 151.11s late 8gBwsSrvZL+1Hg9cKMC0v/zzr8Y Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046094 (already known) 130.92s late wGY/LTzfteifAFQdDbQuhUCZAk4 Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046101 (new)           47.44s late +UOIHbtbGCbAqzy9F/SNPr6E0OY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
Eph 63940/24 11.459 [CHAIN]            info: Received block: 2046101
                  >>> Gossipssub - block 2046101 (already known) 48.29s late +UOIHbtbGCbAqzy9F/SNPr6E0OY Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
Eph 63940/25 0.540 [SYNC]             info: INFURA HEAD 2046105, delay 0.5399999618530273s
                  >>> Gossipssub - block 2046095 (new)           121.27s late 6ljP9gDE6IBgvVnDBPuLchIfQ9Q Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046096 (new)           110.47s late ZCjiZqK/uLnvFvu5bMa0gMz9k4w Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046097 (new)           98.98s late 6w6rIs0Tr15YjpnO6CONAQBPHFY Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046098 (already known) 88.26s late vQ/VxNtn2d/hBMiuzXdPkqiYZHA Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
Eph 63940/25 6.002 []                 info: Synced - block: 2046105   … (empty) - head: 2046101 0x375a…008c - finalized: 2046016 0x6b11…cf1c (63938) - peers: 12
                  >>> Gossipssub - block 2046099 (new)           79.01s late ynTrcsYGjvICvPAlPg6ltVdDRDQ Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046094 (already known) 141.08s late wGY/LTzfteifAFQdDbQuhUCZAk4 Prysm/v1.3.1/6e831920bfa08a300d2786f7039314b03ebbd3ab 16Uiu2HAmLeVegSHYfm9RAebtKetEEGmmebR1VJLBk7KXaYNVsVT6
                  >>> Gossipssub - block 2046100 (already known) 71.15s late 6x6DsvCkteL7pg/cYRHZMyOcXH8 Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046100 (already known) 71.15s late 6x6DsvCkteL7pg/cYRHZMyOcXH8 Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046100 (already known) 71.30s late 6x6DsvCkteL7pg/cYRHZMyOcXH8 rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
Eph 63940/26 0.442 [SYNC]             info: INFURA HEAD 2046106, delay 0.4419999122619629s
                  >>> Gossipssub - block 2046102 (new)           49.37s late jF/7gCQ8O8/dZ18dBtQeMQMJJXM Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
Eph 63940/26 1.400 [CHAIN]            info: Received block: 2046102
                  >>> Gossipssub - block 2046095 (already known) 133.66s late 6ljP9gDE6IBgvVnDBPuLchIfQ9Q Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
                  >>> Gossipssub - block 2046102 (already known) 51.40s late jF/7gCQ8O8/dZ18dBtQeMQMJJXM Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
                  >>> Gossipssub - block 2046096 (already known) 125.16s late ZCjiZqK/uLnvFvu5bMa0gMz9k4w Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
Eph 63940/26 6.001 []                 info: Synced - block: 2046106   … (empty) - head: 2046102 0xb516…8bff - finalized: 2046016 0x6b11…cf1c (63938) - peers: 11
                  >>> Gossipssub - block 2046101 (already known) 68.39s late +UOIHbtbGCbAqzy9F/SNPr6E0OY Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046101 (already known) 68.39s late +UOIHbtbGCbAqzy9F/SNPr6E0OY Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046101 (already known) 68.55s late +UOIHbtbGCbAqzy9F/SNPr6E0OY rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
                  >>> Gossipssub - block 2046095 (already known) 141.15s late 6ljP9gDE6IBgvVnDBPuLchIfQ9Q Prysm/v1.3.1/6e831920bfa08a300d2786f7039314b03ebbd3ab 16Uiu2HAmLeVegSHYfm9RAebtKetEEGmmebR1VJLBk7KXaYNVsVT6
                  >>> Gossipssub - block 2046103 (new)           48.84s late V20kIJsVctEOz2GhqL6me2fUHK8 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
Eph 63940/27 0.858 [CHAIN]            info: Received block: 2046103
Eph 63940/27 1.410 [SYNC]             info: INFURA HEAD 2046107, delay 1.4100000858306885s
                  >>> Gossipssub - block 2046096 (already known) 133.44s late ZCjiZqK/uLnvFvu5bMa0gMz9k4w Prysm/v1.3.1/6e831920bfa08a300d2786f7039314b03ebbd3ab 16Uiu2HAmLeVegSHYfm9RAebtKetEEGmmebR1VJLBk7KXaYNVsVT6
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ [ 'D7+RRGExNLL5q/XORDyARSj8M1k' ]
                  >>> Gossipssub - block 2046107 (new)            2.19s late S8mN8i5Q5syZVy0Jv5Zv8rq7xTo Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ
Eph 63940/27 2.198 [CHAIN]            info: Received block: 2046107
Eph 63940/27 2.200 [CHAIN]           error: Block error slot=2046107, errCode=BLOCK_ERROR_PARENT_UNKNOWN
                  >>> Gossipssub - block 2046097 (already known) 122.45s late 6w6rIs0Tr15YjpnO6CONAQBPHFY Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
Eph 63940/27 2.644 [CHAIN]            info: Received block: 2046106
Eph 63940/27 2.650 [CHAIN]           error: Block error slot=2046106, errCode=BLOCK_ERROR_PARENT_UNKNOWN
                  >>> Gossipssub - block 2046103 (already known) 51.32s late V20kIJsVctEOz2GhqL6me2fUHK8 Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
Eph 63940/27 6.000 []                 info: Synced - block: 2046107   … (empty) - head: 2046103 0x60d2…b69f - finalized: 2046016 0x6b11…cf1c (63938) - peers: 11
                  >>> Gossipssub - block 2046102 (already known) 66.35s late jF/7gCQ8O8/dZ18dBtQeMQMJJXM Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046102 (already known) 66.48s late jF/7gCQ8O8/dZ18dBtQeMQMJJXM Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046102 (already known) 66.48s late jF/7gCQ8O8/dZ18dBtQeMQMJJXM rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
Eph 63940/27 8.762 [CHAIN]            info: Received block: 2046105
Eph 63940/27 8.773 [CHAIN]           error: Block error slot=2046105, errCode=BLOCK_ERROR_PARENT_UNKNOWN
Eph 63940/27 9.164 [CHAIN]            info: Received block: 2046104
Eph 63940/28 0.468 [SYNC]             info: INFURA HEAD 2046108, delay 0.4679999351501465s
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ [ 'qi3RvLEOtEh09u9U2kNFV/8EbEo' ]
                  >>> Gossipssub - block 2046108 (new)            0.84s late eBNqZW1nHsxLuwNThDxvDfAO0yk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ
Eph 63940/28 0.869 [CHAIN]            info: Received block: 2046108
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ [ 'qi3RvLEOtEh09u9U2kNFV/8EbEo' ]
                  >>> Gossipssub - block 2046108 (already known)  2.25s late eBNqZW1nHsxLuwNThDxvDfAO0yk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ
                  >>> Gossipssub - block 2046103 (already known) 62.86s late V20kIJsVctEOz2GhqL6me2fUHK8 rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
                  >>> Gossipssub - block 2046103 (already known) 62.87s late V20kIJsVctEOz2GhqL6me2fUHK8 Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046103 (already known) 62.87s late V20kIJsVctEOz2GhqL6me2fUHK8 Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
Eph 63940/28 6.001 []                 info: Synced - block: 2046108 0x81a3…05a6 - head: 2046108 0x81a3…05a6 - finalized: 2046016 0x6b11…cf1c (63938) - peers: 11
                  >>> Gossipssub - block 2046104 (new)           54.11s late 2KcPLjZYcOkZ5Kk71HpuknBmuKE Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046104 (already known) 56.98s late 2KcPLjZYcOkZ5Kk71HpuknBmuKE Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
                  >>> Gossipssub - block 2046099 (new)           119.37s late ynTrcsYGjvICvPAlPg6ltVdDRDQ Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
                  >>> Gossipssub - block 2046097 (new)           144.28s late 6w6rIs0Tr15YjpnO6CONAQBPHFY Prysm/v1.3.1/6e831920bfa08a300d2786f7039314b03ebbd3ab 16Uiu2HAmLeVegSHYfm9RAebtKetEEGmmebR1VJLBk7KXaYNVsVT6
Eph 63940/29 0.377 [SYNC]             info: INFURA HEAD 2046109, delay 0.377000093460083s
                  >>> Gossipssub - block 2046104 (already known) 63.86s late 2KcPLjZYcOkZ5Kk71HpuknBmuKE Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046104 (already known) 63.86s late 2KcPLjZYcOkZ5Kk71HpuknBmuKE rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
                  >>> Gossipssub - block 2046104 (already known) 64.01s late 2KcPLjZYcOkZ5Kk71HpuknBmuKE Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
Eph 63940/29 6.000 []                 info: Synced - block: 2046109   … (empty) - head: 2046108 0x81a3…05a6 - finalized: 2046016 0x6b11…cf1c (63938) - peers: 13
                  >>> Gossipssub - block 2046105 (new)           56.85s late j7FNunFoXe/SXdYqFj2+oXVVYFI Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046105 (already known) 57.36s late j7FNunFoXe/SXdYqFj2+oXVVYFI Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
                  >>> Gossipssub - block 2046098 (new)           144.37s late vQ/VxNtn2d/hBMiuzXdPkqiYZHA Prysm/v1.3.1/6e831920bfa08a300d2786f7039314b03ebbd3ab 16Uiu2HAmLeVegSHYfm9RAebtKetEEGmmebR1VJLBk7KXaYNVsVT6
Eph 63940/30 0.581 [SYNC]             info: INFURA HEAD 2046110, delay 0.5810000896453857s
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ [ 'ePTyk+FV8suRST1EpySOl0in2vw' ]
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ [ 'ePTyk+FV8suRST1EpySOl0in2vw' ]
                  >>> Requesting peer beacon_block IWANT 16Uiu2HAm73a61x4iG3mXFZmEvavf7RZTsT7v6jwC5TSX84jbLMNZ [ 'ePTyk+FV8suRST1EpySOl0in2vw' ]
                  >>> Gossipssub - block 2046105 (already known) 63.87s late j7FNunFoXe/SXdYqFj2+oXVVYFI Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046105 (already known) 63.87s late j7FNunFoXe/SXdYqFj2+oXVVYFI rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
                  >>> Gossipssub - block 2046105 (already known) 63.87s late j7FNunFoXe/SXdYqFj2+oXVVYFI Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046100 (new)           123.88s late 6x6DsvCkteL7pg/cYRHZMyOcXH8 Prysm/v1.4.4/29d48dfe7eb38959752048aec700f31c3abf7484 16Uiu2HAmH3MSFR4ijLotNJ6qML15uVRqGxkr5S19UUJrp3vmaTSQ
Eph 63940/30 6.001 []                 info: Synced - block: 2046110   … (empty) - head: 2046108 0x81a3…05a6 - finalized: 2046016 0x6b11…cf1c (63938) - peers: 12
                  >>> Gossipssub - block 2046106 (new)           58.02s late H8GMfe+J1R97s9ath42ug2mCkWc Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046106 (already known) 58.33s late H8GMfe+J1R97s9ath42ug2mCkWc Lighthouse/v1.5.0-90d5ab1/aarch64-linux 16Uiu2HAmBi4CiFxB9zQbEx4mkbGiE9bwSaaZ9nGZX8RhRGPXL2fh
Eph 63940/31 0.406 [SYNC]             info: INFURA HEAD 2046111, delay 0.40599989891052246s
Eph 63940/31 6.001 []                 info: Synced - block: 2046111   … (empty) - head: 2046108 0x81a3…05a6 - finalized: 2046016 0x6b11…cf1c (63938) - peers: 12
                  >>> Gossipssub - block 2046106 (already known) 66.46s late H8GMfe+J1R97s9ath42ug2mCkWc Lighthouse/v1.5.0-90d5ab1/x86_64-linux 16Uiu2HAmJj6y2ces4gjhRzPFXBEqRdB6NoKWSh3B1Mgevr41c4WA
                  >>> Gossipssub - block 2046106 (already known) 66.46s late H8GMfe+J1R97s9ath42ug2mCkWc rust-libp2p/0.31.0 16Uiu2HAm3gcqtY6W5717jFQXSfcHkMzGYAag8XuYgcqgDCebeFbz
                  >>> Gossipssub - block 2046106 (already known) 66.46s late H8GMfe+J1R97s9ath42ug2mCkWc Lighthouse/v1.5.0-90d5ab1+/x86_64-linux 16Uiu2HAmCWxz9iXTrstyF1YDbvmpN6ngG2R3cTw6bTAXMLZEN2b8
                  >>> Gossipssub - block 2046107 (new)           57.85s late S8mN8i5Q5syZVy0Jv5Zv8rq7xTo Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
dapplion commented 3 years ago

Found a Prysm peer that's also cronically behind but consistently sending us the blocks afterwards

                  >>> Gossipssub - block 2046150 (new)           71.99s late dDOWwrn6bVegD8guXXDFI9KtNuc Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046151 (new)           70.26s late XimEDW21+1UiG066ahjG74WvSWU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046152 (new)           74.01s late T6x+tkR/M+LTkg5gl5hTS830tH0 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046153 (new)           72.60s late rqQzAIx9jO0e1QRMTUFiYOi3Bqw Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046154 (new)           72.36s late i+7Z5wQeh+p2JRHn6Eg3bZnRfuU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046155 (new)           74.83s late EarIlSVqU39qqlaabbmtuhtRYYY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046156 (new)           75.07s late xQ69VSLHR78HeI+Gxb8pKLuvUnU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046157 (new)           79.33s late uG5OQfy3hbzGyPsYYTsn71UUhPk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046158 (new)           74.48s late KZiQ7CMGnPI/wyX6t2NV8J/yq5o Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046159 (already known) 68.28s late TxctIKCCrZMSw53dYXjygK52mJY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046160 (new)           65.59s late sBpe6fOqJ3bRCWeKGFoxUH9+Mpk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046161 (new)           72.17s late 5FiCA4S6FFUkcnjfgAVvC18APPs Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046162 (new)           71.90s late fKqVbxG8iLJlahYkp7bVSvpM9LU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046163 (new)           71.63s late yQJYS7NKAaLVOMlKp/OyKUN37EM Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046164 (new)           70.84s late +220K8ZHyx2czTJdqKDxN3z/tl0 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046165 (new)           72.33s late Ef5WccIakNcNq0gxEfvAWpP42Ck Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046166 (new)           72.20s late /dtOwzeqeCE2dMtOO6zUd5rbogU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046167 (new)           71.77s late Umg8GYoX3joT9j/YR2fm5AhdmpY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046168 (new)           74.41s late 9gCMIGmpaCxDcJnp9Xpb+ALnpUk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046169 (new)           74.19s late qZvZIeNek+nVYnaha0XbeKpvB0E Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046170 (new)           77.22s late Wg4jOslsCo41rgwaOGT9lALW3FQ Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046171 (new)           77.16s late 9RzsprXE3c63pcD/fpSe/1BnMcM Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046172 (new)           77.72s late 5igKVCXDGCX69ZX5WyINsdzE6tU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046173 (new)           73.33s late Wlbm1i8YyRMlOUnwT5eRQLkG+tg Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046174 (new)           77.14s late ZRELhBj9sncPHrx473RJYfwk84g Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046175 (new)           76.36s late WMtIrahObolMA2QMT8KES1t5yu0 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046176 (new)           68.10s late QdS6YfSm+/yzyNqsKDuNyWYCIg4 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046177 (new)           97.13s late ObRBF1rPDKFa1yLq7XDTk3ocf3o Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046178 (new)           90.50s late kjU9zqxo7iSuPPDZ8HoLizzk6zA Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046179 (new)           86.92s late qWcxGrmJUNwNu4KpW+G/dXiySWM Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046180 (new)           80.71s late M55/FiOmwhXzqu5VkAkwERvnVLs Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046181 (new)           77.84s late J4w4B08joWH5og57sHdqwinp4OY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046182 (new)           72.22s late gibZttNAWpqpD5UK5MsAsfjA1Ic Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046183 (new)           68.32s late mdIMQ6ZB4TIOsFghxXX6pZubYgs Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046184 (already known) 59.89s late ksvtvpRc1cgbKTJCWgGMP+WE+7c Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046185 (already known) 54.85s late Bzn39Q6bAmf5BWl+CzDDQjd/ns0 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046186 (already known) 50.17s late G9qrfsFJrK4fcrKkqas+RjblrXY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046187 (new)           49.49s late Ot3ECOi5ShBTgALbCW95FvxC0d4 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046188 (new)           47.49s late cSTdPgo9frAWqtvhXhkC5MvdiNc Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046189 (new)           47.48s late skQ20bEAVT/BEXsg5N1a6EP9qhg Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046190 (already known) 47.39s late 774bCDk7XDTA3v4vNm68DI+TN2M Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046191 (new)           43.71s late XBxxJ04Fvj7Joh+dk7H4mnUi8/Q Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046192 (new)           42.93s late 0lMuj2i3m2j0036BTNi3LMhrSsM Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046193 (new)           42.28s late 3H++CDaUOhIyp//A4BrpNfiTiGA Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046194 (already known) 42.74s late OLgEiuJ5VtlQkQC0kdF7A6w3msA Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046195 (already known) 43.11s late J+dnndTLBvrGS8Pb+81vCWYYM6s Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046196 (already known) 33.60s late BuDsl67uVkEIS0ZDrsfdWcqJyKs Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046197 (new)           33.99s late hxGmUYpYUXP1iwu/djCVFPtPoJU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046198 (new)           36.92s late 2IDKsHUBZhr5+LMdEx91RcBzWz8 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046199 (new)           37.98s late 4i0tVPpHgNc976kyBjDS0QpnuDk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046200 (new)           35.97s late NggwVknKuie6uCIN+vG59MdeCB0 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046201 (new)           38.72s late 5IQ/lbliiDzh4ihRh7HAWkitynM Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046202 (new)           36.89s late tVS0mWRbTHg1px1AFYluC61iquI Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046203 (already known) 29.46s late 3QbyBPwV97iBLcZIXkf5KOZe8VU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046204 (new)           30.44s late oX6OZA4IEunWy2VqIE07QcVcn3I Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046205 (already known) 24.14s late S0CKRu+OQwI7w/NcQVj21ANmGDc Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046206 (new)           25.26s late d80ThWAjXUD6SO+5+gNhA8ZJyNo Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046207 (already known) 24.65s late oba18p9fYgNOIl4uOnZPsq3/plY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046208 (new)           23.31s late KOg4Lm7YBPbe0UfsEZwYvvSSq9w Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046209 (new)           44.79s late k5tLhlE/nSZfE/LIH5ToE4yXahQ Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046210 (new)           53.45s late njC5steWT6sNKs/QkSKDYpWFTFw Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046211 (new)           46.61s late pUO9LNAFPT8KVAp1TBEgUw+xpHU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046212 (new)           46.04s late s6AJI+xIJy8dq2GrHNHJY3oTO/8 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046213 (new)           65.13s late /N/3jefA931tRwcqUNLSyeg/b0I Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046214 (new)           66.72s late flX0RgeiqnMKjou4BS1cJSUPK+g Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046215 (new)           61.83s late GbHX1aoYZqTBF9Nt+w9dLOg38No Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046216 (new)           61.64s late 6GSTpXDg/CSRQjbh6ltFG3OqOxU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046217 (new)           53.82s late sm75cgQmXYZ8cE209Dlnn3RzmhI Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046218 (new)           51.54s late MY51dwR2TBwVu2SIbQZ1afHez+w Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046219 (new)           44.96s late 0a2Fl2Jh1fjyYC3jL9YyduyZLQ4 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046220 (new)           46.28s late F1m7weDTk5RaDRRFsLD/+nGH/lQ Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046221 (new)           44.84s late 1r3szYpWvupghUtstYmzmnkrE84 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046222 (new)           48.40s late MPRgD2ASxPJP1VObc2bSanKWGZ0 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046223 (new)           51.80s late rbYgTQw/F6krrswcf89x+9lu9PI Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046224 (new)           41.51s late SaXIsvzXpIn/dBX626u0ghOL2fk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046225 (new)           39.02s late J2ij1lF0NVhynthaWyuiGKJTq/4 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046226 (new)           31.83s late 0ooYDYdTx9/aQ4BsItSPB5I0Mjo Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046227 (new)           32.01s late gl659mcGci1dckj+FnVPmRlnrwQ Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046228 (new)           30.32s late conK6rz12BG+Ig0YNicTqCVx76I Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046229 (new)           28.08s late cht28LflHPzuHzq8zn3xZEsPGFQ Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046230 (new)           22.94s late Nq6mZETaw0DlD5OtTNaC2tWks34 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046231 (already known) 21.58s late wfXTHGEeLFs4a7VpqDEL1GyPjrw Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046232 (already known) 16.08s late xifJ4nKiBm5H6RQroXz9QSTujC4 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046233 (already known) 23.54s late HQUsXIuHZ1jrrjijXbu6S4iWTko Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046234 (already known) 18.09s late V+lLbTz+krR3uWl855/TTcmaijc Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046235 (already known) 14.40s late BymdArstf5htrzO4NJfoE2f+yd8 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046236 (already known) 11.32s late nXSqkbLfAiVe+vqUKPWqsCKHAMk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046237 (new)            4.71s late IGZG1PbLTLd9BwaOT+pvcyidzsk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046238 (already known)  4.50s late vH2VsaWdnOUJjPDx5wSKaArimCw Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046239 (new)            1.95s late wrUSlHHHmrBlvMDEpD5hw42VuDI Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046240 (new)            3.33s late nff4zCtqAGTyoih1H94q0PCMMjE Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046241 (already known) 14.06s late 4HdqM4iZ9u/fIcbHs24hyIuOCPU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046242 (already known)  7.34s late 40opchhcqkX9ngQ5M99bigYU+vg Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046243 (already known)  6.31s late PopHF28QrECMe7bgVML2OGsWrJA Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046244 (already known)  3.14s late itzzJNaxzFM5Mu5sey6umwZk0aU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046245 (already known)  4.17s late Wsj6ffNIBkbIwu0Gqi6Mi0OoeGc Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046246 (new)            4.81s late YwMIFuDD6ww6I8Zr52nmp724VNI Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046247 (new)            2.98s late bXXVUZOtmisR5QgchrrwHbbnEmY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046248 (new)            4.51s late iHCXYGpF084vuL8FaPDUYxF+BZ4 Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046249 (new)            9.36s late 66r9Cs7nUVam295Y+gNK0l8uFMY Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046250 (already known) 10.83s late SEJI+Wgf4zWbSHphsb+juQTgrtU Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046251 (new)            5.45s late RRW7wik/XZoi6tlwH2VySfMv7vs Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046252 (new)            0.55s late QcoZbo9Y4vqY7kAcTVEOKhMVpKQ Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046253 (new)            1.76s late VnrbCpTsLccrt0+ahG7/dzDnADk Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
                  >>> Gossipssub - block 2046254 (new)            0.45s late gO2X3JiJMblwPAQP1/Is4juxvYI Prysm/v1.4.2/b3650903ea23b040928732a2bed8b75d584ba903 16Uiu2HAkybvBXJbc9dfmU5spX9yV79Nv44hEQxFiHiEpCmFZR7jA
dapplion commented 3 years ago

It's not true I've added more logging and peers are not being according to their status messages. All peers my node connects to, follow the head very closely (get the head block within a 2-3 seconds margin max). So why are they relaying old blocks?

dapplion commented 2 years ago

When this issue was opened we didn't have enough metrics to properly understand what's going on. Now we receive almost all blocks in time, so no longer an issue.

Moral of the story: invest in metrics kids, not in random logs in the codebase :+1: