search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.52k stars 2.31k forks source link

[Release-1.26] Windows port of K3s agent role #8648

Closed manuelbuil closed 10 months ago

manuelbuil commented 10 months ago

Backport: https://github.com/k3s-io/k3s/issues/7258

mdrahman-suse commented 10 months ago

Validated on release-1.26 branch with commit 4e77f40d

Environment Details

Infrastructure

Node(s) CPU architecture, OS, and Version:

Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1031-aws x86_64)
Windows 2022

Cluster Configuration:

1 Linux server, 1 Linux agent and 1 Windows agent

Config.yaml:

write-kubeconfig-mode: 644
cluster-init: true
node-external-ip: <public-ip>
disable-network-policy: true
token: summerheat

Additional files

pod_client.yaml (clusterIP service) ``` apiVersion: apps/v1 kind: Deployment metadata: labels: app: client name: client-deployment spec: replicas: 2 selector: matchLabels: app: client template: metadata: labels: app: client spec: containers: - image: ranchertest/mytestcontainer imagePullPolicy: Always name: client-curl affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - client topologyKey: kubernetes.io/hostname nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux --- apiVersion: v1 kind: Service metadata: name: client-curl labels: app: client service: client-curl spec: type: ClusterIP selector: app: client ports: - port: 8080 protocol: TCP targetPort: 80 ```
win-app.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: windows-app-deployment spec: selector: matchLabels: app: windows-app replicas: 2 template: metadata: labels: app: windows-app spec: containers: - name: windows-app image: mbuilsuse/pstools:v0.2.0 ports: - containerPort: 3000 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - windows --- apiVersion: v1 kind: Service metadata: labels: app: windows-app-svc name: windows-app-svc namespace: default spec: type: NodePort ports: - port: 3000 nodePort: 30096 name: http selector: app: windows-app ```

Testing Steps

  1. Copy config.yaml 
    $ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
  2. Install k3s on server and agent nodes
  3. Build k3s.exe as suggested here: https://github.com/k3s-io/k3s/pull/7259#issue-1659966888
  4. Copy k3s.exe on Windows node
  5. SSH in to Windows node, activate powershell and run the commands in Windows node 
    New-NetFirewallRule -Name k3s -DisplayName 'k3s' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250
New-NetFirewallRule -Name k3sOverlayTraffic4789UDP -DisplayName "K3s Overlay Traffic 4789 UDP" -Enabled True -Direction Inbound -Protocol UDP -Action Allow -LocalPort 4789
  6. Setup config 
    Set-Content -Path c:/Users/Administrator/k3s-config.yaml -Value @"
token: '<K10-token-from-server>'
server: 'https://<server-public-ip>:6443'
node-external-ip: '<public-ip>'
kubelet-arg=cgroups-per-qos: 'false'
kubelet-arg=enforce-node-allocatable: ''
kubelet-arg=resolv-conf: ''
pause-image: 'mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2022-amd64'
"@
  7. Run k3s.exe using command .\k3s.exe agent --config k3s-config.yaml
  8. Ensure Windows node joins the cluster
  9. Apply the deployments and ensure they are up and running

Validations

$ k3s -v
k3s version v1.26.9+k3s-4e77f40d (4e77f40d)
go version go1.20.8
$ kubectl get node,pods -A -o wide
NAME               STATUS   ROLES                       AGE   VERSION                INTERNAL-IP     EXTERNAL-IP    OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
node/agent1        Ready    <none>                      25m   v1.26.9+k3s-4e77f40d   xxx.xx.xx.153   x.xxx.xx.57    Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1.26
node/ip-ac1f2610   Ready    <none>                      16m   v1.26.9+k3s-4e77f40d   xxx.xx.xx.229   x.xx.xxx.119   Windows Server 2022 Datacenter   10.0.20348.350    containerd://1.7.7-k3s1.26
node/server1       Ready    control-plane,etcd,master   29m   v1.26.9+k3s-4e77f40d   xxx.xx.xx.145   x.xxx.xx.67    Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1.26

NAMESPACE     NAME                                          READY   STATUS            RESTARTS   AGE    IP          NODE          NOMINATED NODE   READINESS GATES
default       pod/client-deployment-7dc79494b4-5rd8x        1/1     Running           0          12m    xx.xx.1.3   agent1        <none>           <none>
default       pod/client-deployment-7dc79494b4-z7rmw        1/1     Running           0          12m    xx.xx.0.9   server1       <none>           <none>
default       pod/windows-app-deployment-68f979c8c4-gzz65   1/1     Running           0          12m    xx.xx.2.3   ip-ac1f2610   <none>           <none>
default       pod/windows-app-deployment-68f979c8c4-kb2jh   1/1     Running           0          12m    xx.xx.2.4   ip-ac1f2610   <none>           <none>
kube-system   pod/coredns-59b4f5bbd5-849kl                  1/1     Running           0          28m    xx.xx.0.6   server1       <none>           <none>
kube-system   pod/helm-install-traefik-crd-z5kcp            0/1     Completed         0          28m    xx.xx.0.5   server1       <none>           <none>
kube-system   pod/helm-install-traefik-jf27q                0/1     Completed         1          28m    xx.xx.0.3   server1       <none>           <none>
kube-system   pod/local-path-provisioner-76d776f6f9-6h5gm   1/1     Running           0          28m    xx.xx.0.4   server1       <none>           <none>
kube-system   pod/metrics-server-68cf49699b-5nr8j           1/1     Running           0          28m    xx.xx.0.2   server1       <none>           <none>
kube-system   pod/svclb-traefik-a131f824-nnf6g              0/2     SysctlForbidden   0          6m1s   <none>      ip-ac1f2610   <none>           <none>
kube-system   pod/svclb-traefik-a131f824-pjs5r              2/2     Running           0          25m    xx.xx.1.2   agent1        <none>           <none>
kube-system   pod/svclb-traefik-a131f824-qn4s4              2/2     Running           0          28m    xx.xx.0.7   server1       <none>           <none>
kube-system   pod/traefik-f75d47c4b-xck74                   1/1     Running           0          28m    xx.xx.0.8   server1       <none>           <none>

Additional Observations