IPv6 peer discovery errors

[garyschulte]

Description

Since #6224, we are seeing a lot of ipv6 protocol errors from ping packet data with an ipv6 from address, e.g.:

VertxPeerDiscoveryAgent | Sending to peer DiscoveryPeer
{ 
  status=bonding,  
 enode=enode://018794617924fd5bc2b207aaa0073279a3bb77e0c5e6285797a262ec7c3dcb05d0bfca64696abc7bdd5dd04730b5bf80d059dc45b9c20fd61557f9ad4716e77c@[2a01:4f9:4b:4f55::2:11]:30303?discport=58267,
  firstDiscovered=1706378307270, 
  lastContacted=0, 
  lastSeen=1706378322275
} failed, 
native error code -97, 
packet: 0x943174d51399b001708462c9a3c361eba66ac98e3505906a1e9090bd9f3b2e6da40e86e7aa16e840b60b2eb56982e620af0aedb39909564c3088db9f1284e8e274d99f1aa06785b8887955cdd4cd9ec7ffae6b1ac05bd9cfe1df9449aee37e770101eb05cb847f000001829d6f829d6fd7902a0104f9004b4f55000000000002001182e39b82765f8465b5448f16,
stacktrace: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: Address family not supported by protocol

Acceptance Criteria

• Survey the current state of IPv6 support in besu
• Decide whether to fix IPv6 discovery or fall back to IPv4 source address when we encounter it

Steps to Reproduce (Bug)

1. run on mainnet for a while with the PeerDiscoveryAgent currently in main
2. observe ipv6 protocol errors in the logs

Expected behavior: [What you expect to happen] No errors responding to peer discoveruy

Actual behavior: [What actually happens] ping packets with ipv6 from addresses fail

Frequency: [What percentage of the time does it occur?] 100%

Logs (if a bug)

2024-01-27 17:58:43.521+00:00 | vert.x-eventloop-thread-1 | WARN  | VertxPeerDiscoveryAgent | Sending to peer DiscoveryPeer{status=bonding, enode=enode://018794617924fd5bc2b207aaa0073279a3bb77e0c5e6285797a262ec7c3dcb05d0bfca64696abc7bdd5dd04730b5bf80d059dc45b9c20fd61557f9ad4716e77c@[2a01:4f9:4b:4f55::2:11]:30303?discport=58267, firstDiscovered=1706378307270, lastContacted=0, lastSeen=1706378322275} failed, native error code -97, packet: 0x943174d51399b001708462c9a3c361eba66ac98e3505906a1e9090bd9f3b2e6da40e86e7aa16e840b60b2eb56982e620af0aedb39909564c3088db9f1284e8e274d99f1aa06785b8887955cdd4cd9ec7ffae6b1ac05bd9cfe1df9449aee37e770101eb05cb847f000001829d6f829d6fd7902a0104f9004b4f55000000000002001182e39b82765f8465b5448f16, stacktrace: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: Address family not supported by protocol

Versions (Add all that apply)

• Software version: 24.1.2-SNAPSHOT
• eth-docker

[siladu]

@matthew1001 any insight into this?

[siladu]

Possibly this issue is limited to docker (e.g. if ipv6 isn't enabled) - could be an ethdocker config issue?

[matthew1001]

I'll have a look in the morning, but it sounds like some nodes set an ipv6 from address but Besu isn't able to create peer connections to them. I'll take a look.

[macfarla]

may be some context here https://github.com/hyperledger/besu/pull/2970

[garyschulte]

This was encountered in eth-docker, so it could well be that ipv6 networking is not supported out-of-the-box. Protocol support would likely be tricky to trap and filter for. We might be better served to have a record type for the host that has the udp source and the ping packet data from address, so we can fall back on connection fail rather than pre-filtering.

@yorickdowne , FYI

[garyschulte]

Thinking about it again, a better alternative would be to detect at startup of the peer discovery agent whether ipv6 is addressable or not and filter accordingly.

[yorickdowne]

This is correct. V6 support in Docker is a bit of a lift. Eth Docker has instructions; I am not sure whether I am doing anything to configure Besu. This requires a) changes to Docker’s config and b) changes to Eth Docker’s config.

https://eth-docker.net/Support/ipv6

[yorickdowne]

I have Besu as “Besu: Possibly no advertisement, no explicit discv5 option”. Happy to change that to “yes” if it has v6 support.

There’s a separate worry about v6 bootnodes. Should likely not be discussed in this issue.

[matthew1001]

It might be that the error handling in https://github.com/hyperledger/besu/blob/3646d78c2ff7c5451318f63dd5f6d8fd5a6962b2/ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/VertxPeerDiscoveryAgent.java#L210 could be extended to put out a friendlier message, like the existing ...this is likely due to ipv6 not being enabled at runtime message.

[siladu]

I've seen reports of more of this since 24.3.0...

besu-mainnet Consensys discord...

Igor Zanella — 15/03/2024 21:05 Hi, I've configured Besu 2 days ago and I let it do the sync, but after the block import is giving the next error VertxPeerDiscoveryAgent | Sending to peer DiscoveryPeer{status=bonding,...... stacktrace: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) fail ed: Address family not supported by protocol What could it be? I'm not running it in a docker, directly on Ubuntu. It seems something involved with IPv6. Thank you

René — 13/03/2024 08:24 I am getting this warning after upgrading to V24.3.0. Is this a problem? 2024-03-12 20:44:04.639+00:00 | vert.x-eventloop-thread-1 | WARN | VertxPeerDiscoveryAgent | Sending to peer DiscoveryPeer{status=bonding, enode=enode://xxx, firstDiscovered=1710281618261, lastContacted=0, lastSeen=1710281618261} failed, native error code -1, packet: 0x141bd9113d611ef2f650e5ed0c6d4453fdd2f2e6415c6803c36358ceb376439789b55d86476e4f06391a1b01c8678dc1b533c9273fc04515895f46d7bee38ae80b9d9369d52a81d6e4e90e816355e3ae1ae0ce682b73953ca57c7ab008486dea0101df05cb847f00000182765f82765fcb84c0a864088276618276618465f0d3e828, stacktrace: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: Operation not permitted

^(likely upgraded from 24.1.0 -> 24.3.0 , not a Docker user)

Think we can disregard this one

as they updated with > after a couple hours of struggle it eventually managed to go back to 25 peers and it seems fine now. I upgrade frequently so I was likely on the previous release before that. #besu-helpdesk Ethstaker discord... (Not sure if related yet) > GGubignou — Yesterday at 19:51 Hi, I updated to the latest Besu via eth-docker and it now seems to struggle finding peers. I was previously at 25 peers constantly but now it oscillates between 8-13 peers. Has anything been changed with regards to port forwarding in the latest version? (FYI my consensus client Teku finds peers as usual and I have otherwise not changed anything to my setup)

[siladu]

Note, I played https://github.com/hyperledger/besu/pull/6728 to move the WARN to DEBUG but wondering if that was the correct thing to do - maybe there is some action they could (should?) take given the following @matthew1001 ?

could be extended to put out a friendlier message, like the existing ...this is likely due to ipv6 not being enabled at runtime message.