search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.52k stars 2.31k forks source link

[meta-issue] Windows port of K3s agent role #7258

Closed seanyen closed 10 months ago

seanyen commented 1 year ago

This is a meta ticket to include the work to bring up K3s agent role for Windows.

Mainly this effort is to contribute the Windows porting effort done in AKS Edge Essentials.

These are the associated pull request or work items:

mdrahman-suse commented 10 months ago

Validated on master branch with commit b8dc9553

Environment Details

Infrastructure

Node(s) CPU architecture, OS, and Version:

Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1031-aws x86_64)
Windows 2022

Cluster Configuration:

1 Linux server, 1 Linux agent and 1 Windows agent

Config.yaml:

write-kubeconfig-mode: 644
cluster-init: true
node-external-ip: <public-ip>
disable-network-policy: true
token: summerheat

Additional files

pod_client.yaml (clusterIP service) ``` apiVersion: apps/v1 kind: Deployment metadata: labels: app: client name: client-deployment spec: replicas: 2 selector: matchLabels: app: client template: metadata: labels: app: client spec: containers: - image: ranchertest/mytestcontainer imagePullPolicy: Always name: client-curl affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - client topologyKey: kubernetes.io/hostname nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux --- apiVersion: v1 kind: Service metadata: name: client-curl labels: app: client service: client-curl spec: type: ClusterIP selector: app: client ports: - port: 8080 protocol: TCP targetPort: 80 ```
win-app.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: windows-app-deployment spec: selector: matchLabels: app: windows-app replicas: 2 template: metadata: labels: app: windows-app spec: containers: - name: windows-app image: mbuilsuse/pstools:v0.2.0 ports: - containerPort: 3000 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - windows --- apiVersion: v1 kind: Service metadata: labels: app: windows-app-svc name: windows-app-svc namespace: default spec: type: NodePort ports: - port: 3000 nodePort: 30096 name: http selector: app: windows-app ```

Testing Steps

  1. Copy config.yaml 
    $ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
  2. Install k3s on server and agent nodes
  3. Build k3s.exe as suggested here: https://github.com/k3s-io/k3s/pull/7259#issue-1659966888
  4. Copy k3s.exe on Windows node
  5. SSH in to Windows node, activate powershell and run the commands in Windows node 
    New-NetFirewallRule -Name k3s -DisplayName 'k3s' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250
New-NetFirewallRule -Name k3sOverlayTraffic4789UDP -DisplayName "K3s Overlay Traffic 4789 UDP" -Enabled True -Direction Inbound -Protocol UDP -Action Allow -LocalPort 4789
  6. Setup config 
    Set-Content -Path c:/Users/Administrator/k3s-config.yaml -Value @"
token: '<K10-token-from-server>'
server: 'https://<server-public-ip>:6443'
node-external-ip: '<public-ip>'
kubelet-arg=cgroups-per-qos: 'false'
kubelet-arg=enforce-node-allocatable: ''
kubelet-arg=resolv-conf: ''
pause-image: 'mcr.microsoft.com/oss/kubernetes/pause:3.8-windows-ltsc2022-amd64'
"@
  7. Run k3s.exe using command .\k3s.exe agent --config k3s-config.yaml
  8. Ensure Windows node joins the cluster
  9. Apply the deployments and ensure they are up and running

Validations

$ k3s -v
k3s version v1.28.2+k3s-b8dc9553 (b8dc9553)
go version go1.20.8
$ kubectl get nodes,pods -A -o wide
NAME                    STATUS   ROLES                       AGE   VERSION                INTERNAL-IP     EXTERNAL-IP      OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
node/agent1             Ready    <none>                      21m   v1.28.2+k3s-b8dc9553   xxx.xx.xx.237   x.xxx.xx.56      Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1
node/server1            Ready    control-plane,etcd,master   31m   v1.28.2+k3s-b8dc9553   xxx.xx.xx.39    xx.xxx.xx.228    Ubuntu 22.04.2 LTS               5.15.0-1031-aws   containerd://1.7.7-k3s1
node/ip-ac1f2610        Ready    <none>                      10m   v1.28.2+k3s-b8dc9553   xxx.xx.x.85     xx.xxx.xxx.138   Windows Server 2022 Datacenter   10.0.20348.350    containerd://1.7.7-k3s1

NAMESPACE     NAME                                          READY   STATUS            RESTARTS   AGE     IP          NODE               NOMINATED NODE   READINESS GATES
default       pod/client-deployment-54fd46977d-9nf2v        1/1     Running           0          3m10s   xx.xx.0.9   server1            <none>           <none>
default       pod/client-deployment-54fd46977d-c4mzv        1/1     Running           0          3m10s   xx.xx.1.3   agent1             <none>           <none>
default       pod/windows-app-deployment-6964ff4fb8-hkrf2   1/1     Running           0          3m10s   xx.xx.2.3   ip-ac1f2610        <none>           <none>
default       pod/windows-app-deployment-6964ff4fb8-v8ll6   1/1     Running           0          3m10s   xx.xx.2.4   ip-ac1f2610        <none>           <none>
kube-system   pod/coredns-6799fbcd5-mdb45                   1/1     Running           0          31m     xx.xx.0.5   server1            <none>           <none>
kube-system   pod/helm-install-traefik-8hd6z                0/1     Completed         1          31m     xx.xx.0.6   server1            <none>           <none>
kube-system   pod/helm-install-traefik-crd-9cz9r            0/1     Completed         0          31m     xx.xx.0.3   server1            <none>           <none>
kube-system   pod/local-path-provisioner-84db5d44d9-vzfzp   1/1     Running           0          31m     xx.xx.0.4   server1            <none>           <none>
kube-system   pod/metrics-server-67c658944b-c8tnv           1/1     Running           0          31m     xx.xx.0.2   server1            <none>           <none>
kube-system   pod/svclb-traefik-72e62b7a-dlbz7              0/2     SysctlForbidden   0          60s     <none>      ip-ac1f2610        <none>           <none>
kube-system   pod/svclb-traefik-72e62b7a-jdgwg              2/2     Running           0          30m     xx.xx.0.7   server1            <none>           <none>
kube-system   pod/svclb-traefik-72e62b7a-qbtds              2/2     Running           0          21m     xx.xx.1.2   agent1             <none>           <none>
kube-system   pod/traefik-7bf7d7576d-6vttx                  1/1     Running           0          30m     xx.xx.0.8   server1            <none>           <none>

Additional Observations