search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.43k stars 2.3k forks source link

Load balancer crashing with IPv6 disabled #10455

Closed lifo9 closed 3 weeks ago

lifo9 commented 1 month ago

Environmental Info:

k3s version v1.29.6+k3s1 (83ae095a)
go version go1.21.11

Node(s) CPU architecture, OS, and Version: 

Linux hostname 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux

Cluster Configuration:

Single node.

Describe the bug:

With IPv6 disabled, svclb-traefik crashes on ContainerCreating with:

/proc/sys/net/ipv6/conf/all/forwarding: no such file or directory

It is caused by securityContext setting:

securityContext:
    sysctls:
    - name: net.ipv6.conf.all.forwarding
      value: "1"

of rancher/klipper container.

It can be fixed by deleting the mentioned net.ipv6.conf.all.forwarding from svclb-traefik daemonset.

Steps To Reproduce:

Expected behavior:

Load balancer does not crash.

Actual behavior:

Load balancer crashes.

Additional context / logs:

brandond commented 1 month ago

This comes from https://github.com/k3s-io/k3s/pull/9963 - which was tested with ipv6 disabled vis sysctl, but apparently not with ipv6.disable=1 in the kernel command line, which I guess removes the net.ipv6 sysctls entirely? I'm confused though since you mention that you have other ipv6-related sysctls set - net.ipv6.conf.all.disable_ipv6=1 for example. Which of these these actually exist on your host with that kernel command line flag set?

Are you able to remove that from your kernel command line as a workaround? The sysctl will come back whenever the loadbalancer daemonset is updated.

lifo9 commented 1 month ago

Thanks for the suggestion, but I'd like to keep IPv6 completely disabled for security reasons (smaller attack surface). Wouldn't it be better to apply that sysctl rule conditionally - check whether IPv6 is enabled?

EDIT: You're right, these sysctl rules are redundant when disabling IPv6 in kernel. But AFAIK, disabling it via kernel is more reliable as the IPv6 module is not loaded at all.

brandond commented 1 month ago

Wouldn't it be better to apply that sysctl rule conditionally - check whether IPv6 is enabled?

Thats what we did previously, but it lead to different problems, as discussed in the issue linked from that PR. I'll do some thinking about better ways to address this. It is unfortunate that we can't ask the container runtime to ignore sysctls that don't exist on a particular node.

brandond commented 1 month ago

I will also note that I have not seen presence of IPv6 on a node as an indicator of additional attack surface; most folks are moving TOWARDS IPv6, not away from it.

mlhynfield commented 1 month ago

Just a quick note since I just happened upon this issue with a new cluster install, there are some cases where IPv6 is disabled at the kernel level by automation or system administration outside the control of an engineer tasked with installing a K3s cluster with load balancer capabilities; in my case I had to scrap the installation and start again with a downgraded version to achieve the level of resilience/stability we required.

provenvelocity commented 1 month ago

I also have this issue upgrading our K3s cluster. Basically, I will disable IPV6 at the cluster. But what is weird is that I set the right settings, and by the time your code makes the daemonset load balancer service, it makes IPV6 and IPV4 entries.

Even with all the setting below are set it still tries to make a ipv6 entry... in the service load balancer

This is the ingress-nginx:

controller:
  kind: DaemonSet
  sysctls:
    "net.ipv4.ip_forward": "1"
  allowSnippetAnnotations: true
  ingressClassResource:
    default: true
  extraArgs:
    default-ssl-certificate: "ingress-nginx/devsecops-tls-secret"
  service:
    ipFamilyPolicy: SingleStack
    externalTrafficPolicy: Local
    external:
      ipFamilies:
      - IPv4
    ipFamilies:
    - IPv4
  admissionWebhooks:
    port: 2443
provenvelocity commented 1 month ago

Also, I can not control my kernel settings.... well I can but they will reset

fmoral2 commented 3 weeks ago

validated on https://github.com/k3s-io/k3s/issues/10567#issuecomment-2277975189 https://github.com/k3s-io/k3s/issues/10568#issuecomment-2277973600 https://github.com/k3s-io/k3s/issues/10569#issuecomment-2277985459 https://github.com/k3s-io/k3s/issues/10570#issuecomment-2277984091