IPv6-only setup fails: cannot configure IPv4 cluster-cidr: no IPv4 CIDRs found

stintel commented 2 years ago

Environmental Info: K3s Version: k3s version v1.22.2+k3s2 (3f5774b4) go version go1.16.8

Node(s) CPU architecture, OS, and Version: Linux k3os-6635 5.4.0-88-generic #99 SMP Tue Oct 5 16:53:38 UTC 2021 x86_64 GNU/Linux

Cluster Configuration: 1 server

Describe the bug: According to https://rancher.com/docs/k3s/latest/en/installation/network-options/#ipv6-only-installation k3s v1.22 and above support IPv6-only setups. Unfortunately passing only an IPv6 subnet to --cluster-cidr fails with a fatal error.

Steps To Reproduce: Run k3s v1.22 or later with the options below:

/sbin/k3s server --node-ip=fe80::5054:ff:fe6b:3304 --cluster-cidr=fc00:6164:6b33:6300::/56 --service-cidr=fc00:6464:6b33:7300::/56

Expected behavior: K3S should start.

Actual behavior:

FATA[0000] cannot configure IPv4 cluster-cidr: no IPv4 CIDRs found

Additional context / logs: The same problem occurs for the --service-cidr option:

# /sbin/k3s server --node-ip=fe80::5054:ff:fe6b:3304 --service-cidr=fc00:6464:6b33:7300::/56
FATA[0000] cannot configure IPv4 service-cidr: no IPv4 CIDRs found

Backporting

[ ] Needs backporting to older releases

brandond commented 2 years ago

Any reason in particular you're using a months-old release of k3s? Can you try with v1.22.10+k3s, or even a 1.23 or 1.24 release? There have been a lot of changes to ipv6 support since last October.

stintel commented 2 years ago

Any reason in particular you're using a months-old release of k3s?

I'm using what's in the latest k3os pre-release.

Can you try with v1.22.10+k3s, or even a 1.23 or 1.24 release? There have been a lot of changes to ipv6 support since last October.

Sure, but if this is a known limitation it would be nice if the documentation page mentions that.

stintel commented 2 years ago

v1.22.8+k3s1 and v1.23.5+k3s1 are the first versions that do not throw the error reported here. They do require adding --disable-network-policy but that's clear from the error message.

As I mentioned earlier, it would be nice if the documentation mentions that. Should we just change the version on https://rancher.com/docs/k3s/latest/en/installation/network-options/#ipv6-only-installation to v1.22.8 and v1.23.5? I can send a PR for that if you want?

j-landru commented 2 years ago

Maybe not related to the issue, but two notes about your IPv6 addressing scheme.

--node-ip=fe80::5054:ff:fe6b:3304 : LLA usage at application level is possible but keeping in mind two things :

these addresses are confined inside broadcast domain (link address) and not route-able,
interface on which LLA is allocated should be specified appending % to the address fe80::5054:ff:fe6b:3304%eth0 for example. In your case maybe, k3s did not silently validate the IPv6 address and fallback IPv4 ?

--cluster-cidr=fc00:6164:6b33:6300::/56 --service-cidr=fc00:6464:6b33:7300::/56 : Private prefix usage, AFAIK RFC4193 specifying ULA address indicates that L bit has to be set to 1 as Zero value is reserved for future usage, so as of today valid private prefix is fd00::/8. fc00::/8 addressing plan is IETF reserved for future usage and should not be used locally.

stintel commented 2 years ago

--node-ip=fe80::5054:ff:fe6b:3304 : LLA usage at application level is possible but keeping in mind two things :

* these addresses are confined inside broadcast domain (link address) and not route-able,

* interface on which LLA is allocated should be specified appending % to the address `fe80::5054:ff:fe6b:3304%eth0` for example.  In your case maybe, k3s did not silently validate the IPv6 address and fallback IPv4 ?

I actually used a global IP in my testing, should have given it more thought before changing it for the report.

--cluster-cidr=fc00:6164:6b33:6300::/56 --service-cidr=fc00:6464:6b33:7300::/56 : Private prefix usage, AFAIK RFC4193 specifying ULA address indicates that L bit has to be set to 1 as Zero value is reserved for future usage, so as of today valid private prefix is fd00::/8. fc00::/8 addressing plan is IETF reserved for future usage and should not be used locally.

Similar story here. I actually started trying without specifying --cluster-cidr or --service-cidr on a host with IPv6-only connectivity. Unfortunately k3os uses connman, which appears to be rather buggy. It does not persist the DUID so after the first reboot the DUID changes, no longer matches what's in the existing lease, and the DHCPv6 server refuses to assign the IP configured in the host reservation. It also doesn't request a hostname from the DHCPv6 server, resulting in a random hostname. I've been trying to find a solution to these issues, but the DHCPv6 options connman request are hardcoded, documentation is slim, and enabling debugging doesn't work 100%, so after wasting a day on that I decided to try another approach. Just didn't give much thought to the ranges.

Either way, thanks for pointing that out. I should also point out that for an IPv6 --service-cidr you cannot use /56:

FATA[0001] apiserver exited: specified --service-cluster-ip-range is too large; for 128-bit addresses, the mask must be >= 108

So on v1.22.8+k3s1, v1.23.5+k3s1 or newer, this does not throw errors and should be a usable example, I believe:

k3s server --cluster-cidr=fd00:6164:6b33:6300::/56 --service-cidr=fd00:6164:6b33:7300::/108

j-landru commented 2 years ago

Unfortunately k3os uses connman, which appears to be rather buggy. Same for me, I didn't find the good way to set IPv6 with connman, I switched back to the native Alpine network settings. See https://github.com/rancher/k3os/discussions/830 and https://framagit.org/-/snippets/6442 for more details.

To give some additional context info, I'm a k3os orphan user, and I try to find an alternative to build an immutable iso file to deploy an IPv6 only k3s cluster. So I left k3os, as this project seems abandoned and switch to elemental-toolkit (former cos-toolkit) trying to build an abcd iso (Alpine based cos derivative), as I'd like to stay with that distro which is not officially supported by elemental-toolkit. (see https://github.com/rancher/elemental-toolkit/issues/1271). This is a WIP in a stand-by state at the moment... but for sure on that context I'll use native open-rc networking service to configure IPv6 network.

before changing it for the report I think the best practice, for that goal is to translate your upper prefix space to 2001:db8::/16 prefix which is dedicated for documentation purpose. All doc, examples, tutorial or howto about IPv6 should be based on that prefix space.

stintel commented 2 years ago

I try to find an alternative to build an immutable iso file to deploy an IPv6 only k3s cluster

Do you happen to hang out on IRC somewhere? I also have some ideas but this issue tracker doesn't seem appropriate to further discuss this.

brandond commented 2 years ago

I'm using what's in the latest k3os pre-release.

I would probably not invest in any new use of K3os at this point.

Do you happen to hang out on IRC somewhere?

Try #k3s on https://slack.rancher.io/

stintel commented 2 years ago

As I mentioned earlier, it would be nice if the documentation mentions that. Should we just change the version on https://rancher.com/docs/k3s/latest/en/installation/network-options/#ipv6-only-installation to v1.22.8 and v1.23.5? I can send a PR for that if you want?

Closing this as the problem does not occur on current version. Please answer the above question and I can do a PR for improving the docs.

Try #k3s on https://slack.rancher.io/

Too bad. I don't do Slack.

k3s-io / k3s

IPv6-only setup fails: cannot configure IPv4 cluster-cidr: no IPv4 CIDRs found #5737