windows support

[perrywang]

there are pretty much customers are using windows at edge side device with installation of labview.

[ibuildthecloud]

This one is tricky. I personally don't know much about Kubernetes on Windows, but Rancher/RKE works on windows so I can find out what needs to be done here. If there's demand, I'd love to do this.

[perrywang]

on windows in general it is using a vm hyperv or vbox to run kubenetes like minikube doing. But minikube is really not production ready and there is a lot of issue and no any customization capability.

[bencompton]

Looking at Microsoft's instructions for joining a Windows node to a cluster, Kubernetes releases have binaries for running Windows worker nodes (kubelet.exe, kube-proxy.exe), but not the control plane. I see there's also an official Windows binary for Flannel, which Microsoft's PS scripts leverage.

I would think a K3s Windows build with only the agent components would be possible. At this point, it looks like Containerd support in Windows is still only in alpha and the official Windows build is still a WIP. Not sure what other dependencies in K3s would be problematic for an agent-only Windows build.

[mortezaalizadeh]

@ibuildthecloud

Hey guys, I just checked the Containerd support for Windows and sounds like it's no longer in alpha state. I am using K3S two of my side projects and supporting Windows on the edge is one of my highest priorities in both projects.

We also don't really need to run the control plane on the edge and Agent support on the Window should be more than enough. I do not know enough about the implementation details of the K3S and K8S, but I am more than happy to help to get the Windows support done.

Is there any progress on this ticket since the last comment posted on 11 April?

[KnicKnic]

my fork can start and join a windows master node. It does appear to have done a few things right, example my kubelet lists container images on my node (I am using docker desktop community). However I don't have networking setup, so the node stays unschedulable.

I would love for someone with more understanding of network configuration to take it further. And give instructions on how to setup networking. @ibuildthecloud do you know anyone that could help here?

cmd I used to get it to "somewhat" startup

.\k3s.exe server -d c:\tmp\k3s --flannel-backend none --docker  --kube-proxy-arg "proxy-mode=userspace" --disable-network-policy

Branch I was using for this experiment. https://github.com/KnicKnic/k3s/tree/k3s_run_windows_node

There are few errors (these related to cni / networking)

W0223 00:59:39.172343   20244 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
E0223 00:59:39.937342   20244 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

And this random one for TLS, have no clue why a cert would be wrong.

2020-02-23 00:54:37.591274 I | http: TLS handshake error from 127.0.0.1:2111: remote error: tls: bad certificate
2020-02-23 00:54:37.602363 I | http: TLS handshake error from 127.0.0.1:2114: remote error: tls: bad certificate

[KnicKnic]

@thxCode @alena1108 @cloudnautique @aiwantaozi pinging some other people in rancher that seemed to have worked on windows integration (for rancher) to see if they could help with setting up windows networking. See my previous comment https://github.com/rancher/k3s/issues/114#issuecomment-590047668 .

[teyc]

@KnicKnic I just bookmarked these:

https://github.com/microsoft/SDN/blob/master/Kubernetes/flannel/register-svc.ps1 a bunch of script for establishing a SDN on Windows.

In particular https://github.com/microsoft/SDN/search?q=cniconfig&unscoped_q=cniconfig Powershell scripts that generate cni configuration.

This is old How To Guide: On-premise Kubernetes for Windows using Flannel (Host-Gateway)

Troubleshooting Kubernetes Networking on Windows Part 1

Also https://github.com/microsoft/SDN/issues/170#issuecomment-548574978

[jiridanek]

Even though I can't run k3s agent on Windows right now, it should be possible to connect a k8s node on Windows to a k3s control node on Linux, yes?

[brandond]

@jiridanek vanilla k8s kubelets cannot connect to k3s servers, as they are missing the k3s-specific bootstrap code.

[maximveksler]

@jiridanek vanilla k8s kubelets cannot connect to k3s servers, as they are missing the k3s-specific bootstrap code.

Any way to work around that? Perhaps a local adapter of some sort?

[andyxhadji]

Very interested in a k3s windows agent as well, has there been any updates here? Happy to contribute here as well

[gruselglatz]

@ibuildthecloud Is there some progress or status on the Windows K3s Agent? Is there some Branch or something to work on to speed things up?

Thanks

[miketzian]

+1 would be keen for this (windows agent + linux server), what is the "k3s-specific bootstrap code" ? (if there's an arch doc, I can look at that)

[MikeTheSnowman]

+1 I too would like to have a package to add in a windows host as a worker node! :)

[moley]

+1 We'd like to use k3s in a windows environment too to handle linux pods

[viceice]

Yeah, joining windows worker nodes would be enough for our use case ❤️

[brandond]

@moley +1 We'd like to use k3s in a windows environment too to handle linux pods

Windows nodes will run Windows containers only. This is true Kubernetes on Windows, not Kubernetes in WSL or anything like that.

[viceice]

Yes, that's what I need. I've some Linux VM for running k3s and want to join some windows VM for running workload which only works on Windows.

[moley]

@moley +1 We'd like to use k3s in a windows environment too to handle linux pods

Windows nodes will run Windows containers only. This is true Kubernetes on Windows, not Kubernetes in WSL or anything like that.

Ok, so we will need a linux vm in our windows environment to get this running? Are there plans to enable this directly in the future?

[lippertmarkus]

Now with Docker Desktop being a paid solution for enterprises this would be valuable for Rancher Desktop.

[brandond]

Rancher Desktop already runs K3s on Windows without any additional work necessary, albeit in a sandbox.

[lippertmarkus]

But no Windows Containers which this issue is about, right?

[MikeTheSnowman]

Hello all. I just wanted to check. Has anyone been able to get a windows worker node to join a k3s cluster?

[Lemonbun]

Would love to see this supported.

[brandond]

We do not currently have any plans to add native Windows support to K3s.

[andruwa13]

In Nomad (https://www.nomadproject.io) the network is normally working on a plugin https://github.com/containernetworking/plugins

[MikeTheSnowman]

We do not currently have any plans to add native Windows support to K3s.

@brandond , pleeeeeease?

[MikeTheSnowman]

Hello. Maybe a different question to ask. But would it be possible to manually add a windows node to a K3s cluster? Has anyone tried? If so, did you encounter any issues?

[pablos91]

Hello. Maybe a different question to ask. But would it be possible to manually add a windows node to a K3s cluster? Has anyone tried? If so, did you encounter any issues?

You could probably experiment with rancher desktop on Windows BUT it will only run Linux containers via WSL2 so it is probably not what everyone is wishing for - a windows container runtime inside a cluster.

[bangzhuzhu]

@wtfacoconut have you tried it?

[MikeTheSnowman]

@wtfacoconut have you tried it?

@bangzhuzhu, are you referring to the recommendation to use WSL2 on Windows? If so then the answer is no, because this would still only run Linux containers.

[bangzhuzhu]

@bangzhuzhu, are you referring to the recommendation to use WSL2 on Windows? If so then the answer is no, because this would still only run Linux containers.

@wtfacoconut I mean manually add a windows node(k8s work node) to a k3s cluster,

[bangzhuzhu]

vanilla k8s kubelets cannot connect to k3s servers, as they are missing the k3s-specific bootstrap code.

@brandond Would mind telling something more about it? Thank you

[tmyroadctfig]

For anyone else for bumps into this thread, Microsoft have started working on AKS Edge, which has an option for k3s and for Windows worker nodes. https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-edge-overview https://github.com/Azure/AKS-Edge

The k3s version of AKS Edge currently spins up a linux hyper-v VM with k3s on it

[brandond]

@tmyroadctfig yes, we're aware. There are ongoing efforts to get this upstreamed.

[viceice]

Windows agent seems to basically work 🎉❤️

• 7258

• 8649

Now there need to be a prebuild k3s.exe on releases so this can be tested more widely. 🙃

[salvioli]

My understanding is that this is not yet supported, is this correct? What is the current status? What are the missing steps to provide windows support?

[mloskot]

By the way, there is an ongoing work enabling Hyper-V-based support for Minikube, automated as much as possible, including non-interactive Windows OS setup for Windows node. Here is the project https://github.com/vrapolinario/MinikubeWindowsContainers Perhaps techniques worked out there may be useful for k3s too.