In dual-stack mode --node-ip to kubelet only have one address

uablrek commented 3 weeks ago

Before creating an issue, make sure you've checked the following:

[X] You are running the latest released version of k0s
[X] Make sure you've searched for existing issues, both open and closed
[X] Make sure you've searched for PRs too, a fix might've been merged already
[X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

vm-001 ~ # uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 6.9.3 k0sproject/k0s#6 SMP PREEMPT_DYNAMIC Fri Jun 14 10:33:26 CEST 2024 x86_64 GNU/Linux
NAME="Xcluster"
VERSION="2024.06.28"
ID=xcluster
PRETTY_NAME="Xcluster"
VERSION_ID="2024.06.28"

Version

vm-001 ~ # k0s version v1.30.2+k0s.0

Sysinfo

`k0s sysinfo`

vm-001 ~ # k0s sysinfo
Total memory: 982.2 MiB (warning: 1.0 GiB recommended)
Disk space available for /var/lib/k0s: 4.8 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.9.3 (pass)
  Max. file descriptors per process: current: 4096 / max: 4096 (warning: < 65536)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /sbin/modprobe (pass)
  Executable in PATH: mount: /bin/mount (pass)
  Executable in PATH: umount: /bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: built-in (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: built-in (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: built-in (pass)
      CONFIG_NF_NAT: built-in (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: built-in (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: built-in (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: built-in (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: built-in (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: built-in (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: built-in (pass)
      CONFIG_LLC: built-in (pass)
      CONFIG_STP: built-in (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

In dual-stack mode only the ipv4 address is specified in the "--node-ip" parameter to kubelet. This causes the node object to have only an ipv4 address.

Steps to reproduce

Start k0s in dual-stack mode
On a worker node do: ps ... | grep kubelet and check the "--node-ip" parameter
Or, do k0s kubectl get node $(hostname) -o json | jq .status.addresses. Only ipv4 is present

Expected behavior

Both ipv4 and ipv6 addresses should be specified in "--node-ip", example:

--node-ip=192.168.1.2,fd00::c0a8:102

Then, when the worker joins, the node object will have both addresses:

# kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "fd00::c0a8:102",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

Actual behavior

Only ipv4 is specified in "--node-ip", and the node object only gets an ipv4 address:

vm-002 ~ # k0s kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

Screenshots and logs

No response

Additional context

I install with k0sctl.

k0scat.yaml

```yaml apiVersion: k0sctl.k0sproject.io/v1beta1 kind: Cluster metadata: name: k0s-cluster spec: k0s: version: $__k0sver config: apiVersion: k0s.k0sproject.io/v1beta1 kind: Cluster metadata: name: k0s spec: api: k0sApiPort: 9443 port: 6443 installConfig: users: etcdUser: etcd kineUser: kube-apiserver konnectivityUser: konnectivity-server kubeAPIserverUser: kube-apiserver kubeSchedulerUser: kube-scheduler konnectivity: adminPort: 8133 agentPort: 8132 network: kubeProxy: disabled: false mode: iptables provider: calico calico: mode: "bird" envVars: IP_AUTODETECTION_METHOD: "interface=eth1" IP6_AUTODETECTION_METHOD: "interface=eth1" kuberouter: autoMTU: true mtu: 0 peerRouterASNs: "" peerRouterIPs: "" podCIDR: 10.244.0.0/16 serviceCIDR: 10.96.0.0/16 dualStack: enabled: true IPv6podCIDR: "$PREFIX:0af4:0:0/108" IPv6serviceCIDR: "$PREFIX:0a60:0:0/108" podSecurityPolicy: defaultPolicy: 00-k0s-privileged storage: type: etcd telemetry: enabled: false hosts: - ssh: address: 192.168.1.1 user: root port: 22 keyPath: /root/.ssh/id_dropbear_ssh role: controller+worker installFlags: - --disable-components=konnectivity-server noTaints: true os: alpine privateInterface: eth1 privateAddress: 192.168.1.1 k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64 - ssh: address: 192.168.1.2 user: root port: 22 keyPath: /root/.ssh/id_dropbear_ssh role: worker os: alpine privateInterface: eth1 privateAddress: 192.168.1.2 k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64 - ssh: address: 192.168.1.3 user: root port: 22 keyPath: /root/.ssh/id_dropbear_ssh role: worker os: alpine privateInterface: eth1 privateAddress: 192.168.1.3 k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64 - ssh: address: 192.168.1.4 user: root port: 22 keyPath: /root/.ssh/id_dropbear_ssh role: worker os: alpine privateInterface: eth1 privateAddress: 192.168.1.4 k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64 ```

twz123 commented 3 weeks ago

I don't think k0s passes any --node-ip flags to kubelet at all. According to the CLI flag's description, it defaults to IPv4. That's what you're seeing in your cluster, I suppose.

  --node-ip string                                           IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass '::' to make it prefer the default IPv6 address rather than the default IPv4 address.

You could pass your own values via --kubelet-extra-args=--node-ip=... in spec.hosts[*].installFlags for the time being.

/cc @ncopa WDYT about this?

uablrek commented 3 weeks ago

I don't think k0s passes any --node-ip flags to kubelet at all.

@twz123 You are right. I specified privateAddress:. If I remove that, no "--node-ip" is passed to kubelet.

vm-002 ~ # ps www | grep kubelet
  464 root     1267m S    /var/lib/k0s/bin/kubelet --root-dir=/var/lib/k0s/kubelet --config=/var/lib/k0s/kubelet-config.yaml --kubeconfig=/var/lib/k0s/kubelet.conf --v=1 --cert-dir=/var/lib/k0s/kubelet/pki --containerd=/run/k0s/containerd.sock --runtime-cgroups=/system.slice/containerd.service

But for dual-stack it seems that you must specify "comma-separated dual-stack IP addresses" to get it right:

vm-002 ~ # k0s kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

(ipv6 is missing)

twz123 commented 3 weeks ago

If it's connected to privateAddress, then it's probably something in k0sctl, not in k0s itself ... Indeed, it is: https://github.com/k0sproject/k0sctl/blob/v0.18.1/pkg/apis/k0sctl.k0sproject.io/v1beta1/cluster/host.go#L320-L327

Not sure what to do here, since the IPv6 is not part of the k0sctl config at all. /cc @kke How would k0s or k0sctl know which IP to use if it isn't specified? Sure, there could be some magic auto-detection, but less magic is usually preferable. Would it make sense to make privateAddress a list and accept multiple IPs, or is that a bad idea? :thinking:

uablrek commented 3 weeks ago

You could pass your own values via --kubelet-extra-args=--node-ip=... in spec.hosts[*].installFlags for the time being.

Yes, that works :smiley:

vm-002 ~ # k0s kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "fd00::c0a8:102",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

I removed privateAddress: and instead specified:

    installFlags:
    - "--kubelet-extra-args=--node-ip=192.168.1.2,fd00::192.168.1.2"

If privateAddress: is only used for passing an address to kubelet via --node-ip, it should take comma-separated dual-stack IP addresses, same as kubelet.

For a better user experience, k0sctl may specify the first IPv4 and the first global IPv6 for "privateInterface:" as --node-ip if dual-stack is enabled.

This problem should be documented in any case.

I am happy with this work-around, and you may close this issue if you like, or keep it for reference.

ncopa commented 3 weeks ago

k0sproject / k0sctl