RFC: Support IPv6 for network isolated add-ons

[pvizeli]

Context

Docker IPv6 support does not really exist and not usable because most users have an IPv6 host network from the provider without routing. Docker itself is driven by the big cloud provider which has very limited IPv6 support.

However, our system should work in IPv6 and IPv4 in the same way.

Decision

We add a new plugin called ipv6 which implement NAT6 -> https://github.com/robbertkl/docker-ipv6nat Next, we need to take an IPv6 address alias over NetworkManager and attach it to the DNS plugin for support ipv6 DNS server.

[jhampson-dbre]

Since this was originally opened, the docker-ipv6nat repo mentioned above has pinned an issue considering deprecation due to recent progress in Docker for built in IPv6 NAT support.

[pvizeli]

Nice, so we can update libnetwork on our OS

[TimoFriedri]

Hi,

the issue mentioned here is probably the reason for addons like Wireguard not to work. I have an IPv6 only provider. I can transmit to the Wireguard plugin from my phone (I see that on the sent and received data) BUT WG cannot answer via IPv6. So the phone never receives anything. --> no handshake

So I think this issue is highly relevant for a growing amount of people, since Dual Stack-Lite is more and more common.

[pvizeli]

It comes in 1-2 years. Docker/Moby is used and developed by some big cloud company that has self not good IPv6 support. But of the curse, the need comes and they start to fix their product which allows us to improve our software.

State today, IPv6 only installation doesn't work, you need to make dual-stack support on your network and tun it to an ipv4 gateway.

[TimoFriedri]

Uff, that's a bummer, Basically means an extra ~5Euro per month for a lot of people to their provider.

I think I have to run wireguard outside the docker environment directly on the machine, not nice, but managable. Thanks for the feedback.

[ShadowJonathan]

Linking https://github.com/home-assistant/supervisor/issues/1405 for extra context.

+1 to this, its 2022 now, does docker properly support ipv6 networking now? Kubernetes does, so at least that should correlate.

[tmoore22]

+1 to this, its 2023 now, does docker properly support ipv6 networking now?

[g00gleit]

+1 to this I recently switched to T-Mobile and it's annoying that I can't setup IPV6 to work with the Wireguard server on home assistant.

[bikeshedder]

Not being able to use IPv6 is a real bummer to me as well. I'm seriously considering kicking HASSOS in favor of a custom container installation or setting up my own reverse proxy.

I do love the easy updates HASSOS provides. That's the only reason that I haven't switched, yet.

The lack of IPv6 support almost looks like a planned oversight in order to push the Home Assistant Cloud service which shouldn't be a requirement to access HASS installations on a IPv6 only network.

[agners]

Not being able to use IPv6 is a real bummer to me as well.

@bikeshedder just to be clear: Home Assistant OS supports IPv6 just fine! You can also connect to Home Assistant using IPv6 just fine! Home Assistant Core uses host network, and we do support IPv6 on the main network via Network Manager.

This issue is about supporting IPv6 for add-ons which do not run on host network. The Supervisor (or rather the Docker container engine) provides an isolated network for add-ons. This isolated network uses IPv4 currently. This typically is no big deal as almost all setups still use dual-stack (IPv4 and IPv6). So, if necessary, the add-ons can connect to the Internet via IPv4 still. This whole issue is really only relevant if you run a IPv6 only network.

[bikeshedder]

The problem I run into is the carrier grade NAT from my provider. Yes, add-ons can still access IPv4 services. It cannot use services which are only exposed via IPv6 though. This breaks the DuckDNS add-on if I want to expose my HASS Installation via IPv6 as it can't reliably detect the public facing IPv6 address.

Dedicated public facing IPv4 addresses are becoming less common in Germany as the IPv4 address pools are well known to be depleted for quite some time now.

IPv6 was introduced 27 years ago and is the only solution to end that CGN madness once and for all.

What needs to be done in order to enable IPv6 for add-ons?

[TimoFriedri]

This whole issue is really only relevant if you run a IPv6 only network.

Many do unintentionally, because the provider only allows full IPv6 and crippled IPv4.

My only solution was to move Wireguard out of the Home Assistant Machine and run it separately.

[agners]

Many do unintentionally, because the provider only allows full IPv6 and crippled IPv4.

I guess you talk about CGNAT? That use case is handled well by the current implementation since the add-ons only reach out via IPv4, which works through CGNAT.

My only solution was to move Wireguard out of the Home Assistant Machine and run it separately.

Yeah that add-on might benefit from full IPv6. However, I guess what you want is to have it reachable by IPv6? In that case we would need to assign the network public IPv6 addresses, which not in all IPv6 setups might be necessary.

Btw, the Wireguard add-on decides to not run on host network, which makes it more a Wireguard "client" (I know there is no such thing, but from an architecture more like a typical VPN client in the sense of not be reachable from the outside). That is a decision decision of that add-on.

[cleopold73]

Ran into this issue while trying to get more stuff working over ipv6 in my internal network. In my case it was the unifi addon talking to the unifi controller. It still work since I'm dual stack, but i was a bit disappointed that add-ons can't talk outbound to anything via ipv6.

[ShadowJonathan]

I'm going to assume that HAOS runs docker, in which case you can go looking here for ipv6 support: https://docs.docker.com/config/daemon/ipv6/

However, it's tagged as "experimental", which doesn't inspire much confidence

[Giga-Pudding]

According to the official documentation, assigning private IPv6 addresses (ULA) to the docker containers is possible. IPv6 NAT is also possible.

Here's a good summary about the current status: https://github.com/robbertkl/docker-ipv6nat/issues/65#issuecomment-1568033924 Although... this comment is now 8 months old. IPv6 support may be even better today.

Does anyone see a reason, why it shouldn't be possible to use IPv6 inside HA OS docker containers?

However, it's tagged as "experimental", which doesn't inspire much confidence

Yes, it still requires the experimental flag, but this parameter is enabled in HA OS anyway.

[deviantintegral]

I think it's (unfortunately) still important to consider and have checks for broken IPv6 connectivity with fallbacks for addons.

A case in point: Bell Canada's mobile internet for rural service has been provisioning IPv6 addresses for years. While those addresses are global addresses, they don't actually support routing traffic over IPv6. This recently came up setting up Home Assistant Cloud, where setup fails until you disable IPv6 in HAOS.

This isn't an issue with typical desktop or mobile systems because they have some sort of IPv6 connectivity test and fallback, such as with https://en.wikipedia.org/wiki/Happy_Eyeballs.

[bdraco]

aiohttp doesn't support happyeyeballs yet, but will in 3.10 it will after https://github.com/aio-libs/aiohttp/pull/7954

Enabling ipv6 is likely not a good idea if there are any aiohttp requests until we upgrade to aiohttp 3.10+ (not yet released) since there will be unexpected breakage otherwise.

[agners]

I think it's (unfortunately) still important to consider and have checks for broken IPv6 connectivity with fallbacks for addons.

Well but the same could be said for Home Assistant Core and Supervisor itself. :man_shrugging:

In fact, most add-ons probably don't use aiohttp. However, if whatever they use implements happy eyeballs support is a different question of course :see_no_evil:

I do understand the concern, but I think this mainly adds the requirement that users should be able to disable the feature if necessary. We probably could tie it to the host network configuration (e.g. if there is no host network IPv6 configured, then we'd also not provide IPv6 addresses for add-ons). Having an internet routable IPv6 address is presumably the minimum requirement for NAT66 to be useful :thinking:

[agners]

I've started a related discussion about macvlan support in Supervisor, see https://github.com/home-assistant/architecture/discussions/1034.

As the title of this discussion says, this issue is about IPv6 for network isolated add-ons. Introducing macvlan support would add another way for add-ons to get IPv6 support without using host network. I also think that macvlan would work very well together with IPv6: macvlan interfaces need IP addresses assigned. This is much simpler in IPv6 world thanks to SLAAC.

[mraerino]

quick note that i got this working on an out-of-the-box Homeassistant OS install. Maybe it can be an inspiration for how to solve it natively:

Enabling IPv6 for addon containers

You need to run all of these commands on a native SSH session. The SSH addon won't work. 1. Stop supervisor: `systemctl stop hassos-supervisor` 2. Stop all running containers: ``` for ID in $(docker ps --format '{{ .ID }}'); do docker stop $ID; done ``` 3. Remove all containers: ``` for ID in $(docker ps -a --format '{{ .ID }}'); do docker rm $ID; done ``` 4. Remove the `hassio` network ``` docker network rm hassio ``` 5. Re-create the `hassio` network with IPv6 support enabled ``` docker network create hassio -d bridge --ipv6 --subnet --subnet 172.30.32.0/23 --ip-range 172.30.33.0/24 --gateway 172.30.32.1 --opt com.docker.network.bridge.name=hassio ``` **Remember to insert your ULA net!** 6. Reboot!

This had the effect of giving all addon containers an IPv6 address that works for outgoing connections.

The only trouble i had was that the core_nginx_proxy addon does not listen on IPv6 by default, which i was able to work around but would need to be fixed before IPv6 is rolled out to everyone (edit: see PR https://github.com/home-assistant/addons/pull/3475).

The nice consequence is that now trusted networks work with IPv6 clients when using the nginx addon.

[agners]

@mraerino I guess this is more or less what #3780 implements then, no?

What I am a bit worried is how add-ons typically behave in environments which do not have (global) IPv6 connectivity. From what I understand this adds NAT66. And the approach makes software inside the add-ons think IPv6 is available, in any case. Techniques such as Happy eyeballs should make sure to fallback to IPv4 quickly, but this might lead to slowdowns or issues with happy eyeballs is not beeing used.

[Giga-Pudding]

Why not let the users decide on their own? Let's have a switch in the network config: "enable upstream IPv6 support (full IPv6 support by ISP is mandatory)"

This switch should be disabled by default, if introduced via Update (for existing installations). This switch should be enabled by default, for new installations (if people experience issues, they can disable it).

If this function were always disabled by default, it would not help to promote the use of IPv6. Other OSes also have IPv6 enabled by default, so i think it should be enabled by default in Home Assistant too.

[codyc1515]

I would strongly encourage enabling by default. However, is there a security risk from this? Many people using IPv4 rely on NAT as a form of firewall - IPv6 does not (usually) have this.

[Giga-Pudding]

In this case, private IPv6 addresses are used (unique local addresses). Docker performs a NAT66, so what happens is basically the same as with IPv4.

Exposed Containers do not exist by default. The containers shouldn't be reachable from outside, without proper iptables rules.

[mraerino]

@agners i think the proposal we've landed on in this issue seems reasonable. what is the process for making it happen?

[Giga-Pudding]

@mraerino can you do me a favor and test something? What happens, if IPv6 is enabled for containers like you did it, but the supervised host does not have a working IPv6 upstream for some reason? How does Docker behave in that case? Will Containers still think there's a working IPv6 upstream and try to send traffic via IPv6 into the internet? Or is Docker clever enough to remove the IPv6 default gateway for the containers?

Can you set the IPv6 configuration in your running Home Assistant OS to "disabled" and see what happens?

[agners]

I have the same concern as @Giga-Pudding has here: How do the add-ons behave in absence of IPv6 support on the host side...

@mraerino I don't quite understand your setup at this point. In your case, what is "your ULA net"? Is that the same ULA you use on your regular LAN? So you essentially you bridge the IPv6 network so the add-on are directly in your LAN? If so then there is no NAT66 in play on the Home Assistant side? But how can you then reach the public internet, is NAT66 active on your router? :thinking:

[mraerino]

In your case, what is "your ULA net"? Is that the same ULA you use on your regular LAN? So you essentially you bridge the IPv6 network so the add-on are directly in your LAN?

no, i just created a fresh ULA prefix (using https://unique-local-ipv6.com/) and used it. NAT66 in Docker is used. if i had prefix delegation enabled in my router i might have used that to get public addresses, but for now i use NAT.

[Giga-Pudding]

@mraerino can you test this?

@mraerino can you do me a favor and test something? What happens, if IPv6 is enabled for containers like you did it, but the supervised host does not have a working IPv6 upstream for some reason? How does Docker behave in that case? Will Containers still think there's a working IPv6 upstream and try to send traffic via IPv6 into the internet? Or is Docker clever enough to remove the IPv6 default gateway for the containers?

Can you set the IPv6 configuration in your running Home Assistant OS to "disabled" and see what happens?

[Giga-Pudding]

@agners there's no progress here. Can we at least introduce full IPv6 support via NAT66 as experimental feature (opt-in)?

[benniju]

Is there any progress on this? I would really like IPv6 support for different reasons for example for monitoring IPv6 addresses. (https://github.com/hassio-addons/repository/issues/618) I think it's pretty much a no go to not properly support IPv6 in 2024.

[Meister1977]

+1 I need ipv6 support for multiple reason. Would be nice if HA enable this feature... Please... Maybe, you can create 2 docker networks? Keep the hassio and and a hassiov4v6 and I as a user can assign the new network? It would be backward compatible?

[lorinc-ambrozy-work]

+1 Please, we need this feature.

[Eisbergsalat]

I recently got infected by DS-lite.... And now I even have to change my Uptime Kuma from HA to docker? 😢

I hope everyone else can stand tall during the DS-lite pandemic. In hope you wont get infected 🫡

[gtxaspec]

+1 Its 2024, IPv6 is needed by default.