Feature Request: IPv6 support

fireduck64 commented 3 years ago

Description

Many networks now have native IPv6 support. In my experience, IPv6 is great for p2p applications because often hosts on ipv4 nat networks will have their own globally routable IPv6 address. This can allow for running multiple of the name service behind a single IPv4 address or help with uPNP/port forwarding.

Present Behaviour

As far as I have seen, all the lighthouse modes seem to open only bind to ipv4 and only advertise ipv6 to peers. I could be wrong, I haven't dug into the code, just looked with 'lsof'

Expected Behaviour

In an ideal world, the when doing advertisements on a p2p network the implementation would scan the IPv6 addresses bound to interfaces and each one that is in the global ipv6 space, consider it for inclusion in the advertisement to peers.

https://www.juniper.net/documentation/en_US/junos/topics/concept/subscriber-management-dual-stack-ipv6-address-types.html

Security Implications

There is one downside. Many sys admins are used to thinking in terms of IPv4 and port forwarding. So if the IPv6 address of their service nodes are advertised on the P2P network, those admins might be surprised that many of the services they are running might then be exposed to the world.

In general the IPv6 address space, even on a private residential internet provider has a large enough address space where it is nearly impossible to guess IPv6 addresses on that network unless they are in some way shared (like connections from the host via IPv6 or the host advertising itself on some P2P protocol). (Each connection having 2^64 address makes it hard to guess valid ones)

For this reason, it might make sense to make IPv6 support opt-in rather than on by default.

AgeManning commented 3 years ago

Yep, this has been on my radar for a while. See: https://github.com/sigp/lighthouse/issues/1617.

In principle, all our networking sub-protocols support ipv6 but there are complexities that require care and thought when permitting ipv6 throughout the codebase. The security implication you mentioned is one example.

In regards to advertising the address to peers, we can allow a user to advertise ipv6 address and ports through their ENR, but logic around setting both ipv6 and ipv4 socket addresses in an ENR and preferencing one over the other when attempting a connection has not been considered (We by default preference ipv4).

The default behaviour (which I assume is applicable for most users) is to automatically discover the external socket address and advertise that for peers to connect to. The way this works is by sending UDP packets to known peers on the network. When enough of them respond with the same source socket address, we consider that our external socket address and advertise that. The issue here is that if we are only running ipv6 and our peers are running ipv4, I imagine we send outbound packets via ipv4 and they will respond with our ipv4 external socket address and this is what we will advertise.

I think the solution is to have ipv6 support but also fallback to ipv4 for maximal connectivity. This involves some preferencing in our discovery protocol and we've delayed this task until after mainnet launch. Keep an eye out on the issue I've linked for progress.

andrejpodzimek commented 3 years ago

Here are a few observations (and also follow-ups on my comments under #1617 and openethereum/openethereum#156):

In openethereum, listening on :: works as expected, i.e., IPv6 “wildcard” sockets exhibit their default dual-stack behavior. So one can see tcp6 peer connections in netstat -atpn, such as:
```
Proto  Recv-Q  Send-Q  Local Address  Foreign Address  State        PID/Program name
tcp6    14480       0  x.x.x.x:40056  y.y.y.y:30303    ESTABLISHED  931780/openethereum
tcp6        0       0  x.x.x.x:30303  z.z.z.z:5635     ESTABLISHED  931780/openethereum
```
Notice tcp6 versus IPv4 addresses. So this (sort of) works as intended.

Quite unfortunately though, many outbound connections show up as tcp, not tcp6, which means that despite the IPv6 listening capabilities, some (:question:) lookups and some outbound connections insist on IPv4-only sockets instead of using backward-compatible dual stack ones.
In lighthouse, however, listening on :: does not work. :warning: It can be configured and lighthouse starts, but no connections are established. I'm not entirely sure what the root cause could be. Either\ (a) lighthouse is unable to set up outbound connections in that mode (and make itself known) or\ (b) IPV6_V6ONLY might be set on sockets somewhere under the hood and there are (almost) no IPv6-capable peers to talk to.

So in lighthouse, setting :: and hoping for the best is not a viable option / workaround.

This^^^ difference in behavior is surprising, because both openethereum and lighthouse are written in Rust and both use a similar set of libraries. However, I haven't been able to spot any obvious causes of such a difference (half-working listen-only IPv6 in openethereum versus completely broken IPv6 in lighthouse).

For the record, here's an idea from the Cardano world. They split the flags into --host-addr and --host-ipv6-addr to make an explicit IPv6 / IPv4 distinction. The only drawback is that the default dual-stack support (IPv6 sockets for both IPv4 and IPv6) cannot be used, which leads to an equivalent of IPV6_V6ONLY IPv6 sockets with additional IPv4-only sockets. However, there are also a few advantages:

One can listen on different ports for IPv4 and IPv6. (This is not used in cardano-node, but it's possible.)
One can configure IPv6-only nodes (and get rid of all IPv4 mess), which is great for legacy-free networks.

My only gripe would be the flag names: IMO, --host-addr should have required IPv6 by default and the other flag should have been called --host-addr-ipv4-get-rid-of-this-soon or some such.

fireduck64 commented 2 years ago

I just want to note that since we can have the 'vc' use a list of beacon nodes, this is more of an issue now. Some of my backup beacons are on distant networks and only have ipv6 direct access so not being able to have a beacon answer on an ipv6 port makes my failover more complex.

AgeManning commented 2 years ago

Ok. I dont think this is an overly large task to complete.We should hopefully have some time to address this and I'll increase this in priority and get proper IPv6 support.

divagant-martian commented 2 years ago

I'll work on this

xrchz commented 2 years ago

IPv6 for the HTTP Beacon Node API server is also important for my setup. I'd be interested to help in a small way if I can.

macladson commented 2 years ago

@xrchz I've raised an issue to track progress here: https://github.com/sigp/lighthouse/issues/3103

gitcoinbot commented 2 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 1000.5003 USD (1000.0 USD @ $1.0/USD) attached to it.

If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
- Want to chip in? Add your own contribution here.
- Questions? Checkout Gitcoin Help or the Gitcoin's Discord
- $5,051,065.63 more funded OSS Work available on the Gitcoin Issue Explorer

gitcoinbot commented 2 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 264 years, 7 months from now. Please review their action plans below:

1) divagant-martian has been approved to start work.

Hi, I've actually been working on this for the last couple of weeks. First step is adjusting the discovery protocol to find ipv6 peers, and then req/resp. I think I'm close to finishing the discovery work, but testing is still needed

Learn more on the Gitcoin Issue Details page.

one-three-three-seven commented 2 years ago

I'm the one who created the bounty, I would really like full IPv6 support. I would like to run Lighthouse within IPv6-only environments without additional setup, no manual peering or similar. This issue and https://github.com/sigp/lighthouse/issues/1617 being fixed would be a huge progress.

@divagant-martian I have seen you already announced to work on this issue and made progress. If you want to apply for the bounty, I would be happy to approve you. I can give it to someone who can help you, if you don't want it. A short answer what the current situation/plan is would be interesting.

It's the first time I made a Gitcoin bounty, if there is something wrong about it, please tell me.

divagant-martian commented 2 years ago

Hi @one-three-three-seven just answered via gitcoin. I've been working on this on the side lately and would be happy to claim and finish the work. I also aim for full support. As I said in the gitcoin answer, this should be more or less a 3 steps work. First, discovery, then the rpc, and finally some additional servers. This is in order of more to less work required. The discovery progress I expect to be able to finish in a week or two. The rpc, I'm not sure but I expect it to be easier. Also new to gitcoin :)

AgeManning commented 1 year ago

This has been completed. We should now have full support for ipv6. Our mainnet bootnodes have been updated which allow onboarding of ipv6 nodes.

sigp / lighthouse