INFURA / devp2p-network

Tools for analyzing the DevP2P network
Apache License 2.0
32 stars 18 forks source link

Lots of Subprotocol errors #6

Open kshinn opened 5 years ago

kshinn commented 5 years ago

I'm seeing a lot of peers getting removed because of a subprotocol error.

infura:main 213.136.76.42: eth.63,eth.62 +44ms
infura:main 213.136.76.42 SUBPROTOCOL_ERROR +7ms
infura:main 46.101.160.39: eth.63,eth.62 +10ms
infura:main 46.101.160.39 SUBPROTOCOL_ERROR +16ms
  infura:main 199.247.3.3 SUBPROTOCOL_ERROR +7ms
  infura:main 89.34.99.3: eth.62,eth.63,par.1,par.2,pip.1 +65ms
  infura:main 35.234.97.198: eth.62,eth.63,par.1,par.2,par.3,pip.1 +27ms
  infura:main 51.68.172.123 SUBPROTOCOL_ERROR +14ms
  infura:main 51.140.82.22 SUBPROTOCOL_ERROR +3ms
  infura:main 159.65.200.185: eth.63,eth.62 +39ms
  infura:main 137.117.158.210 SUBPROTOCOL_ERROR +22ms
  infura:main 35.234.97.198 undefined +82ms
  infura:main 37.139.23.36: eth.63,eth.62 +11ms
  infura:main 176.223.131.39: eth.63 +85ms
  infura:main 37.139.23.36 SUBPROTOCOL_ERROR +168ms
  infura:main 41.185.97.129: eth.63,eth.62 +2ms
  infura:main 176.223.131.39 SUBPROTOCOL_ERROR +1ms
  infura:main 41.185.97.129 SUBPROTOCOL_ERROR +688ms
  infura:main 206.189.68.191 DISCONNECT_REQUESTED +3s
  infura:main 89.34.99.3 DISCONNECT_REQUESTED +1s
  infura:main 159.65.200.185 SUBPROTOCOL_ERROR +95ms

I've checked the handshake and it looks like the library is doing that properly on the peer connect. I suspect we are seeing a number of missing nodes because of these protocol mismatches.

kshinn commented 5 years ago

The ethereumjs code seems to have a problem parsing the body and assumes the protocol is out of spec or an unknown variant.

egalano commented 5 years ago

It'd be interesting to see as much information as possible from the client connection that is reporting the subprotocol error. Strange custom clients, older versions maybe? The DISCONNECT_REQUESTED could also be coming from a peer that does not have available peering slots

kshinn commented 5 years ago

From what I can tell the disconnect request are part of the normal protocol. I've been seeing them after a successful status call. There is an explicit peering table full error I see as well. There may be value in tracking these in the db as well.

On Fri, Dec 28, 2018, 11:21 PM E.G. Galano <notifications@github.com wrote:

It'd be interesting to see as much information as possible from the client connection that is reporting the subprotocol error. Strange custom clients, older versions maybe? The DISCONNECT_REQUESTED could also be coming from a peer that does not have available peering slots

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/INFURA/devp2p-network/issues/6#issuecomment-450472185, or mute the thread https://github.com/notifications/unsubscribe-auth/AACEUXoU_QiNvsAbGX29xhicb9xKSgIEks5u9xgMgaJpZM4ZkcVu .

kshinn commented 5 years ago

Got a little more information. The reason we are getting a protocol mismatch looks like a network ID mismatch between the Peer status message and the devp2p status message. It ends up throwing an assertion message.

Devp2p client message:

  infura:main 52.3.158.184: eth.62,eth.63 +216ms
  infura:p2p [ <Buffer 3f>,
  infura:p2p   <Buffer 01>, <-- Mainnet
  infura:p2p   <Buffer 04 00 00 00 00>,
  infura:p2p   <Buffer d4 e5 67 40 f8 76 ae f8 c0 10 b8 6a 40 d5 f5 67 45 a1 18 d0 90 6a 34 e6 9a ec 8c 0d b1 cb 8f a3>,
  infura:p2p   <Buffer d4 e5 67 40 f8 76 ae f8 c0 10 b8 6a 40 d5 f5 67 45 a1 18 d0 90 6a 34 e6 9a ec 8c 0d b1 cb 8f a3> ] +195ms

Looking at the wire protocol the first byte is the protocol version (63) The second is the NetworkID. The 01 above is Mainnet, what the devp2p client is expecting.

Peer status


infura:p2p [ <Buffer 3f>,
  infura:p2p   <Buffer 04>, <-- Rinkby
  infura:p2p   <Buffer 65 51 35>,
  infura:p2p   <Buffer de ed 9b 12 f5 b3 bd bb 51 a9 e6 65 04 00 37 6d 11 0a 90 76 09 96 5e cc d6 b8 61 f0 b7 ea 53 03>,
  infura:p2p   <Buffer 63 41 fd 3d af 94 b7 48 c7 2c ed 5a 5b 26 02 8f 24 74 f5 f0 0d 82 45 04 e4 fa 37 a7 57 67 e1 77> ] +2ms
  infura:main 52.3.158.184:30303 (Peer Error) AssertionError [ERR_ASSERTION]: NetworkId mismatch: 01 / 04 +1ms
  infura:main 52.3.158.184 SUBPROTOCOL_ERROR +2ms
kshinn commented 5 years ago

Posted for reference

No. hex Network Notes
0 0x00 Olympic Ethereum public pre-release testnet
1 0x01 Frontier Homestead, Metropolis, the Ethereum public main network
1 0x01 Classic the (un)forked public Ethereum Classic main network, chain ID 61
1 0x01 Expanse an alternative Ethereum implementation, chain ID 2
2 0x02 Morden the public Ethereum testnet, now Ethereum Classic testnet
3 0x03 Ropsten the public cross-client Ethereum testnet
4 0x04 Rinkeby the public Geth PoA testnet
8 0x08 Ubiq the public Gubiq main network with flux difficulty chain ID 8
42 0x2A Kovan the public Parity PoA testnet
77 0x4D Sokol the public POA Network testnet
99 0x63 Core the public POA Network main network
100 0x64 xDai the public MakerDAO/POA Network main network
401697 0x62121 Tobalaba the public Energy Web Foundation testnet
7762959 0x7F740F Musicoin the music blockchain
61717561 0x3ADBC39 Aquachain ASIC resistant chain