`k0s worker` on Windows is unable to validate the Kubernetes CA

NiklasRosenstein commented 1 year ago

Before creating an issue, make sure you've checked the following:

[X] You are running the latest released version of k0s
[X] Make sure you've searched for existing issues, both open and closed
[X] Make sure you've searched for PRs too, a fix might've been merged already
[X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

For the Windows Worker Node: Windows 2022 Datacenter Edition

Linux 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version

v1.26.2+k0s.1

Sysinfo

`k0s sysinfo`

Machine ID: "797a672b57c33565656933b08dad29c890bbd308c7270a8a6ab6902bcf14d13d" (from machine) (pass)
Total memory: 62.5 GiB (pass)
Disk space available for /var/lib/k0s: 853.7 GiB (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.15.0-69-generic (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  Executable in path: modprobe: /usr/sbin/modprobe (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (pass)
    cgroup controller "memory": available (pass)
    cgroup controller "devices": available (assumed) (pass)
    cgroup controller "freezer": available (assumed) (pass)
    cgroup controller "pids": available (pass)
    cgroup controller "hugetlb": available (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

Running the k0s worker on Windows doesn't trust the Kubernetes CA. Wouldn't establishing that trust be embedded in the token? Could it be an issue because I have the Linux nodes (controller and other workers) managed by k0sctl?

$ .\k0s-v1.26.2+k0s.1-amd64.exe worker --cidr-range=10.244.0.0/16 --cluster-dns 10.96.0.10 --cri-socket=docker:tcp://127.0.0.1:2375 --api-server=https://mycontrollernode:6443 H4sIAAAAAAAC/2... -d
time="2023-03-30 15:56:09" level=debug msg="Starting debug server" debug_server=":6060"
time="2023-03-30 15:56:09" level=debug msg="no config file given, using defaults"
time="2023-03-30 15:56:09" level=debug pre-flight-check=machine-id property="\"efcda2534a4552818d8f9775eb4f08747686e84e9012f0163e3a8bb6b3b23404\" (from machine)"
time="2023-03-30 15:56:09" level=warning pre-flight-check=memory property="Total memory detection unsupported on this platform"
time="2023-03-30 15:56:09" level=warning pre-flight-check="disk:C:\\var\\lib\\k0s" property="Disk space detection unsupported on this platform"
time="2023-03-30 15:56:09" level=info msg="No cached API server addresses found"
time="2023-03-30 15:56:09" level=debug msg="Failed to load configuration for worker profile in attempt #1, retrying after backoff" error="Get \"https://100.74.212.124:6443/api/v1/namespaces/kube-system/configmaps/worker-config-default-1.26\": x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes-ca\")"
time="2023-03-30 15:56:09" level=debug msg="Failed to load configuration for worker profile in attempt #2, retrying after backoff" error="Get \"https://100.74.212.124:6443/api/v1/namespaces/kube-system/configmaps/worker-config-default-1.26\": x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes-ca\")"

Steps to reproduce

Setup a controller with k0sctl
Try to join a Windows worker to the cluster

Expected behavior

No response

Actual behavior

No response

Screenshots and logs

No response

Additional context

No response

NiklasRosenstein commented 1 year ago

Using Calico instead of kube-router now as I'm supposed to 😄 Now I'm getting this error that I'm unsure how to resolve:

time="2023-03-31 18:15:42" level=warning msg="exit status 1" component=kubelet.exe
time="2023-03-31 18:15:42" level=info msg="respawning in 5s" component=kubelet.exe
time="2023-03-31 18:15:47" level=error msg="Probing components" component=prober
time="2023-03-31 18:15:47" level=info msg="Restarted (4)" component=kubelet.exe
time="2023-03-31 18:15:47" level=info msg="Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI." component=kubelet.exe stream=stderr
time="2023-03-31 18:15:47" level=info msg="E0331 18:15:47.984733    5964 run.go:74] \"command failed\" err=\"failed to parse kubelet flag: unknown flag: --cni-conf-dir\"" component=kubelet.exe stream=stderr

Another interesting issue is that in this state I can't seem to Ctrl^C the k0s worker process.

NiklasRosenstein commented 1 year ago

After removing C:\bootstrap.ps1, C:\CalicoWindows and C:\var and rerunning k0s.exe worker (with a fresh token, just in case), I get

time="2023-03-31 18:38:31" level=warning pre-flight-check=memory property="Total memory detection unsupported on this platform"
time="2023-03-31 18:38:31" level=warning pre-flight-check="disk:C:\\var\\lib\\k0s" property="Disk space detection unsupported on this platform"
time="2023-03-31 18:38:31" level=info msg="Bootstrapping kubelet client configuration using win-6sehifo1fki as node name"
time="2023-03-31 18:38:32" level=info msg="No cached API server addresses found"
time="2023-03-31 18:38:32" level=info msg="initializing OCIBundleReconciler"
time="2023-03-31 18:38:32" level=info msg="initializing Kubelet"
time="2023-03-31 18:38:32" level=info msg="initializing KubeProxy"
time="2023-03-31 18:38:32" level=info msg="initializing CalicoInstaller"
time="2023-03-31 18:38:32" level=info msg="Staging 'C:\\var\\lib\\k0s\\bin\\kubelet.exe'"
time="2023-03-31 18:38:32" level=info msg="Staging 'C:\\var\\lib\\k0s\\bin\\kube-proxy.exe'"
time="2023-03-31 18:38:32" level=info msg="initializing Status"
time="2023-03-31 18:38:32" level=info msg="initializing Autopilot"
time="2023-03-31 18:38:32" level=info msg="Listening address C:\\var\\lib\\k0s\\run\\status.sock" component=status
Error: can't get calico-kube-config: unexpected response status: 403 Forbidden

And after another invokation I get the above error message again.

jnummelin commented 1 year ago

I believe you stumbled upon https://github.com/k0sproject/k0s/issues/2460

NiklasRosenstein commented 1 year ago

Do I read correctly from that issue that Windows support in K0s with Kubernetes 1.24+ is completely broken?

jnummelin commented 1 year ago

Do I read correctly from that issue that Windows support in K0s with Kubernetes 1.24+ is completely broken?

Pretty much yes. 😢

Can you share some light into what is your use case for running Windows containers?

We have a plan in place to get the Windows side back on track. The obvious big challenge for us is to come up with a solution for CI test automation (GH Action) so that we can actually test it too on each PR.