Closed vitaliy-leschenko closed 2 years ago
@jonaskello you get a "nat network already exists error", that's different to what OP got. Try to remove the already existing one:
Get-HNSNetwork | Remove-HNSNetwork
@lippertmarkus Yes you are correct, when we switched to hostprocess it is a different error from OP. Actually we got one step further by deleting the cni config file created by Install-Containerd.ps1 like you do here:
Now flannel and kube-proxy starts and the workload container gives error about invalid architecture on Server 2022. Will try it on 2019 also.
@lippertmarkus Yes you are correct, when we switched to hostprocess it is a different error from OP. Actually we got one step further by deleting the cni config file created by Install-Containerd.ps1 like you do here:
Now flannel and kube-proxy starts and the workload container gives error about invalid architecture on Server 2022. Will try it on 2019 also.
Hi @jonaskello where you able to make it work, I am facing the same issue with ContainerD and Flannel on Windows Node
@K31D Yes we have it working now.
Although it is not possible to run 2019 images on Server 2022 worker nodes but I found out this is a limitation of windows itself.
@jonaskello may I ask you how or what was the problem ? Cause I am trying with WS2019+ContainerD 1.5.9 and Flannel (vxlan)
@K31D If I recall correctly deleting the CNI config as per my comment above solved it for us.
@K31D if you are using the hostprocess containers you need to use containerd 1.6+ and k8s 1.22+ with the feature flag enabled https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/#before-you-begin
Thank you @jsturtevant, no they are "regular" containers. Unfortunately, pods are keeping getting the Nat network instead of the Pod Cidr. We want to try by either using DockerD or WS2022, we are forced to use Vxlan cause we are on Azure IaaS VM.
yea the scripts for containerd had a bug that wasn't fixable with this solution. The only way to get flannel working with containerd is to run flanneld as a windows service on the host and only configure the flannel network not the nat config that the scripts set up.
we are forced to use Vxlan cause we are on Azure IaaS VM.
There is no reason to only use vxlan here. in AKS and capz the vms created as Nodes are IaaS VMs. We have gotten all of the various cni's working including ones that use l2bridge configurations.
thanks again @jsturtevant I've tried to find a doc on how to install it as a service, the only guide I found is this one but it's 3yrs old also the official guide on how to add windows node was deleted from kubernetes doc
we know this type of installation guide is missing, it has been challenging to create one due to differences in setups/networks and lack of parity for things like privileged containers. We have discussed as a sig-windows to potentially fill this gap with some tutorials but have limited folks to dedicate to it.
That basics in those scripts you link are essentially still the same as they were 3 years ago unless you are trying to use cni's and kubeproxy as hostprocess containers. Calico also has some good docs and scripts that mostly just work: https://projectcalico.docs.tigera.io/getting-started/windows-calico/quickstart
For anyone interested I made a gist with the scripts that we have working to install windows worker nodes on our on-prem cluster that was setup using kubeadm. We use this on freshly installed WS2019 Core machines. It uses ContainerD and flannel with host process containers. I think I got the original scripts from a gist @lippertmarkus had and then we tweaked them a bit.
Thank you again both! You have been very helpful, will try the @jonaskello script
@jonaskello @lippertmarkus We have a need for improving our getting started for windows: https://github.com/kubernetes-sigs/sig-windows-tools/issues/217
Would either of you be interested in helping improve our getting started story with the work you've already done?
@jsturtevant sure. Next to the sources you listed in https://github.com/kubernetes-sigs/sig-windows-tools/issues/217#issuecomment-1192804934 there's also https://github.com/microsoft/Windows-Containers/tree/Main/helpful_tools by @brasmith-ms
I think it would be better to decide on one installation method we want to provide as "official" and then add all features and configuration options to that. I created the containerd-installer because we wanted an easy way to install containerd and PowerShell scripts can't be used for a WinGet target. So if we want to move forward with that installer, we could transfer the repo to sig-windows and I'm happy to create a WinGet package for it. The same could be done fore kubelet and kubeadm.
Installation could look like
winget install containerd --override "--include-cni"
winget install kubelet # installs kubelet as a service
winget install kubeadm
Related: https://github.com/microsoft/winget-cli/discussions/2361#discussioncomment-3211512
For other CNIs and kube-proxy we should work on making the installation via hostprocess containers the official path as that's similar to how it works on Linux and we wouldn't need windows specific docs.
@K31D do you manage to make the cluster work? with script from @jonaskello i get an error from kube-proxy-windows pod due to missing cni configuration, the crictl info also report missing configuration. maybe i am missing something here
@ghbeta yes, unfortunately, it was not me but a colleague. They said they just followed @jonaskello instructions. Our final environment was 3 control plane ubuntu 18.04 2 windows server 2019 (patched to the most recent configuration) + Flannel as CNI
@K31D @ghbeta Hi, I have followed the script of @jonaskello and my windows worker node successfully connected to the linux master (CentOS7). pods are successfully running in the cluster. but i am not able to access via NodePort service. connection is refusing.
Any idea ?
For anyone having a similar issue while running containerd and using host-gw
flannel, I found that restarting the box will cause the containerd
network to drop out of scope. This can be remedied with these 2 steps:
New-HnsNetwork -Type NAT -Name nat
- create the missing network, as NAT is somehow deleted during rebootGet-HnsEndpoint | ? { $_.Name -eq 'cbr0_ep' } | Remove-HnsEndpoint
- remove an endpoint that I think was still tied to the old NAT.
Step 2 I got from #94 after getting the error message rpc error: code = Internal desc = could not create IP forward entry: The object already exists.
Describe the bug Flannel and kube-proxy pods do not start.
To Reproduce
Node successfully joined to cluster:
Pods stuck on ContainerCreating
Error message:
Expected behavior flannel and kube-proxy started
Kubernetes: