lightserv: repeated attempts to `dyndial` other-genesis LES clients

veox commented 6 years ago

System information

Geth version: 1.8.4-stable (installed via Ubuntu PPA package) OS & Version: Ubuntu 16.04.4 LTS (Xenial Xerus)

Expected behaviour

On LES handshake failure, especially due to genesis block mismatch, no repeat dial should be attempted, for a prolonged period of time.

Actual behaviour

A dyndial to the same ~~incompatible~~ other-genesis peers is attempted repeatedly.

EDIT: To clarify, IMO peer exchange with other ether-networks is nice, and might come in handy when sharding; but hammering known-different peers with connections is not.

Steps to reproduce the behaviour

geth run as:

/usr/bin/geth --metrics --datadir /home/geth/.ethereum --cache 4096 --txpool.globalslots 16384 --txpool.pricelimit 31337000 --lightserv 50 --lightpeers 725 --maxpeers 750

Enable les debug messages in console:

debug.vmodule("les=4")

Observe log.

Backtrace

Is this necessary to demonstrate this particular issue?..

Here's the relevant log output:

DEBUG[04-18|12:19:00] Light Ethereum peer connected            id=65d01089d2e42275 conn=dyndial    name=Geth/v1.7.3-stable/linux-amd64/go1.9.2
DEBUG[04-18|12:19:01] Light Ethereum handshake failed          id=65d01089d2e42275 conn=dyndial    err="Genesis block mismatch - dde1c9209b315ed2 (!= d4e56740f876aef8)"
DEBUG[04-18|12:19:13] Light Ethereum peer connected            id=bbc3b33e4a8d6c04 conn=dyndial    name=Geth/v1.8.0-unstable-908faf8c/linux-amd64/go1.9.2
DEBUG[04-18|12:19:13] Light Ethereum handshake failed          id=bbc3b33e4a8d6c04 conn=dyndial    err="Genesis block mismatch - a4293815e8a06b09 (!= d4e56740f876aef8)"
DEBUG[04-18|12:19:25] Light Ethereum peer connected            id=47692f11d9f82fd4 conn=dyndial    name=Geth/v1.8.3-unstable-933972d1/linux-amd64/go1.10
DEBUG[04-18|12:19:25] Light Ethereum handshake failed          id=47692f11d9f82fd4 conn=dyndial    err="Genesis block mismatch - a373ec376981118b (!= d4e56740f876aef8)"
DEBUG[04-18|12:19:31] Light Ethereum peer connected            id=65d01089d2e42275 conn=dyndial    name=Geth/v1.7.3-stable/linux-amd64/go1.9.2
DEBUG[04-18|12:19:31] Light Ethereum handshake failed          id=65d01089d2e42275 conn=dyndial    err="Genesis block mismatch - dde1c9209b315ed2 (!= d4e56740f876aef8)"
DEBUG[04-18|12:19:55] Light Ethereum peer connected            id=47692f11d9f82fd4 conn=dyndial    name=Geth/v1.8.3-unstable-933972d1/linux-amd64/go1.10
DEBUG[04-18|12:19:55] Light Ethereum handshake failed          id=47692f11d9f82fd4 conn=dyndial    err="Genesis block mismatch - a373ec376981118b (!= d4e56740f876aef8)"
DEBUG[04-18|12:20:01] Light Ethereum peer connected            id=65d01089d2e42275 conn=dyndial    name=Geth/v1.7.3-stable/linux-amd64/go1.9.2
DEBUG[04-18|12:20:02] Light Ethereum handshake failed          id=65d01089d2e42275 conn=dyndial    err="Genesis block mismatch - dde1c9209b315ed2 (!= d4e56740f876aef8)"
DEBUG[04-18|12:20:05] Light Ethereum peer connected            id=bbc3b33e4a8d6c04 conn=dyndial    name=Geth/v1.8.0-unstable-908faf8c/linux-amd64/go1.9.2
DEBUG[04-18|12:20:05] Light Ethereum handshake failed          id=bbc3b33e4a8d6c04 conn=dyndial    err="Genesis block mismatch - a4293815e8a06b09 (!= d4e56740f876aef8)"
DEBUG[04-18|12:20:25] Light Ethereum peer connected            id=47692f11d9f82fd4 conn=dyndial    name=Geth/v1.8.3-unstable-933972d1/linux-amd64/go1.10
DEBUG[04-18|12:20:26] Light Ethereum handshake failed          id=47692f11d9f82fd4 conn=dyndial    err="Genesis block mismatch - a373ec376981118b (!= d4e56740f876aef8)"
DEBUG[04-18|12:20:32] Light Ethereum peer connected            id=65d01089d2e42275 conn=dyndial    name=Geth/v1.7.3-stable/linux-amd64/go1.9.2
DEBUG[04-18|12:20:32] Light Ethereum handshake failed          id=65d01089d2e42275 conn=dyndial    err="Genesis block mismatch - dde1c9209b315ed2 (!= d4e56740f876aef8)"

(It's the same three peers.)

veox commented 6 years ago

This might actually be two issues collated:

not saving the ~~"other-genesis" information after a handshake failing for that reason~~ reason why a dyndial failed, and subsequently trying to re-dial other-genesis peers as often as same-genesis ones;
trying to dyndial "light" peers in the first place (on my lightserv, near-100% of les peers are inbound).

ligi commented 6 years ago

cc @zsfelfoldi

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

veox commented 5 years ago

In trinity, a peer database is maintained, and past failures - say, "genesis mismatch" - are recorded. Past failures add a "timer" to skip re-dialing that particular peer; the nature of the failure determines the timer's duration. Wrong network/failure have the timer set to one day.

zsfelfoldi commented 5 years ago

I am looking at it now. The "server pool" (the peer selection logic of clients) is indeed a bit dumb, one of its problems is that currently it does not distinguish between reasons of failure like "no free slots at the moment" and permanent incompatibility.

zsfelfoldi commented 5 years ago

This issue seems to be primarily an issue with the (soon to be replaced) experimental discv5 discovery. Sometimes somehow discv4 and discv5 peers get mixed up so ETH nodes sometimes try to dial random LES nodes too in the hope of establishing an ETH connection. This will be fixed by the new discv5 and I don't think it is a serious issue in itself. The fact that these peers are repeatedly hammered is bad but it is a general p2p issue unrelated to LES.

karalabe commented 5 years ago

Based on the logs, these seem to originate from a full node dialing a light client (or rather dialing a random node and matching the les protocol capability). To fix this, we need to make the discovery protocol a bit smarter, which we're currently doing with ENRs. It's a known issue (same happens during finding testnet peers for example). There's no inherent issue, just a protocol limitation. Will be fixed soon-ish.

ethereum / go-ethereum