search

paritytech / substrate-telemetry

Polkadot Telemetry service
GNU General Public License v3.0
299 stars 208 forks source link

The backend reports duplicate peers. #497

Open DamianStraszak opened 1 year ago

DamianStraszak commented 1 year ago

On https://telemetry.polkadot.io there are lots of duplicate entries shown, with the copies all being stuck on the same block 11857185 -- probably marks the moment when something bad happened to the backend.

The issue is present on all chains, not only polkadot. Attaching screenshot

Screenshot from 2022-09-08 12-12-37

jsdw commented 1 year ago

There is a small chance that this will be resolved by https://github.com/paritytech/substrate-telemetry/pull/504.

I did spend a bunch of time carefully digging through the code to see why this might be the case and nothing stands out at the moment. There is logic in place to handle various things being disconnected and restarted and it seems robust offhand. I believe we do have malicious connections/nodes and what you saw might have been an instance of it.

If it continues to be an issue after #504 I'll eventually dig more into it, but my best guess so far is malicious or broken nodes, since stale nodes and service disconnects all lead to fairly aggressive cleanups.