search

KWasm / kwasm-node-installer

Installs KWasm on Kubernetes nodes.
Apache License 2.0
35 stars 13 forks source link

v0.3.0 Test Matrix #43

Closed 0xE282B0 closed 1 year ago

0xE282B0 commented 1 year ago

This is the kwasm-node-installer v0.3.0 release issue. The PR https://github.com/KWasm/kwasm-node-installer/pull/42 updates deislabs/containerd-wasm-shims to v0.9.0 and containerd/runwasi to the latest master (as soon as the next release is out, we will pin the version to it).

The latest shims introduced some changes, so we need to test these shims and the installer on all supported platforms. The test documentation consists of three parts: cluster setup, test setup, and test result. The Kind test can ne used as template. Feel free to test K8s distributions you are familiar with and post the test documentation as a reply to this issue.

K8s Distribution WasmEdge Wasmer WasmTime Lunatic Spin Slight WWS
Kind ✅ ✅ ✅ ✅ ✅ ✅ ✅
MicroK8s ✅ ✅ ✅ ✅ ✅ ✅ ✅
Minikube ✅ ✅ ✅ ✅ ✅ ✅ ✅
CoLima ✅ ✅ ✅ ✅ ✅ ✅ ✅
Killercoda ✅ ✅ ✅ ✅ ✅ ✅ ✅
AWS EKS ✅ ✅ ✅ ✅ ✅ ✅ ✅
Azure AKS
Civo K3s ✅ ✅ ✅ ✅ ✅ ✅ ✅
Digital Ocean K8s ✅ ✅ ✅ ✅ ✅ ✅ ✅
GCP GKE
Namespace.so ✅ ✅ ✅ ✅ ✅ ✅ ✅
0xE282B0 commented 1 year ago

Kind

Recording: https://asciinema.org/a/SvNWIgRo0xdKQEkCj0cyQ293b

Prerequisites

Setup cluster

# Crate cluster
kind create cluster --image kindest/node:v1.28.0

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached.. Notes: There is a known issue where the sidecar does not start on the first run after installation. If you delete it, it will stay in Terminating forever, but the new Pod will start and both containers will work fine.

❯ kubectl get pod                                
NAME                             READY   STATUS        RESTARTS   AGE
lunatic-demo-6475554875-zbrwh    2/2     Running       0          73s
wasm-slight-6467bcc5bc-gx29z     2/2     Running       0          73s
wasm-spin-74c4cf5c77-4slq4       2/2     Running       0          73s
wasm-wws-888f6bc4b-qj6rx         2/2     Running       0          73s
wasmedge-demo-5ff758d79-b5v6j    2/2     Running       0          73s
wasmer-demo-857f947cb7-8qx5g     1/2     Terminating   2          73s
wasmer-demo-857f947cb7-xqw4s     2/2     Running       0          12s
wasmtime-demo-56c78ddd95-x6wmc   2/2     Running       0          73s
0xE282B0 commented 1 year ago

Colima

Recording: https://asciinema.org/a/nF5m7fHgdr3Bq8Qdj1scy66Jq

Prerequisites

Setup cluster

# Crate cluster
colima start -k -r containerd --kubernetes-version v1.28.1+k3s1

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached. Notes:

Only runwasi shims become ready:

❯ kubectl get pod                                                                                     
NAME                             READY   STATUS             RESTARTS      AGE
wasmer-demo-857f947cb7-lr7rk     2/2     Running            0             2m39s
wasmtime-demo-56c78ddd95-8jsbz   2/2     Running            0             2m39s
wasmedge-demo-5ff758d79-7v4rx    2/2     Running            0             2m39s
wasm-slight-6467bcc5bc-pqt2x     0/2     CrashLoopBackOff   8 (47s ago)   2m39s
wasm-wws-888f6bc4b-gnzz4         0/2     CrashLoopBackOff   8 (43s ago)   2m39s
lunatic-demo-6475554875-wqphh    0/2     CrashLoopBackOff   8 (30s ago)   2m39s
wasm-spin-74c4cf5c77-m5z5x       0/2     CrashLoopBackOff   3 (11s ago)   2m39s
0xE282B0 commented 1 year ago

Namespace.so

Recording: https://asciinema.org/a/QnreeaoqsldJWUi6ULsvpykT0

Prerequisites

Setup cluster

cat > kwasm.json << EOF
{"disks":[{"name":"kwasm","image":"ghcr.io/kwasm/kwasm-node-installer:pr-42"}],"containerd_shims":[{"name":"lunatic","runtime_type":"/disk/kwasm/assets/containerd-shim-lunatic-v1"},{"name":"slight","runtime_type":"/disk/kwasm/assets/containerd-shim-slight-v1"},{"name":"spin","runtime_type":"/disk/kwasm/assets/containerd-shim-spin-v1"},{"name":"wws","runtime_type":"/disk/kwasm/assets/containerd-shim-wws-v1"},{"name":"wasmedge","runtime_type":"/disk/kwasm/assets/containerd-shim-wasmedge-v1"},{"name":"wasmer","runtime_type":"/disk/kwasm/assets/containerd-shim-wasmer-v1"},{"name":"wasmtime","runtime_type":"/disk/kwasm/assets/containerd-shim-wasmtime-v1"}]}
EOF
nsc create --experimental_from kwasm.json
nsc kubeconfig write 1rq3tlt0mfpu6
export KUBECONFIG=~/Library/Caches/ns/tmp/kubeconfig/973245096.yaml

Test execution

Namespace.so clusters can't restart containerd. When containerd is terminated the whole cluster is terminated. But they have a parameter to customize containerd config before startup.

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached. Notes: There is a known issue where the first Pod does not start on the first run after installation. If you delete it, it will stay in Terminating forever, but the new Pod will start and both containers will work fine.

❯ kubectl get pod                            
NAME                             READY   STATUS        RESTARTS      AGE
wasmer-demo-558c75c49f-p7sf6     2/2     Running       0             30s
wasmtime-demo-6979d575-48nq4     2/2     Running       0             30s
lunatic-demo-767757cf9-2x55j     2/2     Running       0             30s
wasmedge-demo-7fb59fffdc-46k4l   2/2     Running       0             30s
wasm-spin-85bfb69795-8q877       2/2     Running       0             30s
wasm-slight-5646fd7cd5-swtft     2/2     Running       1 (26s ago)   30s
wasm-wws-c875b9664-wf84x         0/2     Terminating   0             29s
wasm-wws-c875b9664-8btjt         2/2     Running       0             5s
Mossaka commented 1 year ago

There is a known issue where the sidecar does not start on the first run after installation. If you delete it, it will stay in Terminating forever

Would you please link the issue (if the issue was not created, raise an issue to upstream) here for reference?

Mossaka commented 1 year ago

Disk space for / is at 100% after copying the shims

Oh no, the symbol stripping work @jprendes did might help to mitigate the disk pressure issue! https://cloud-native.slack.com/archives/C04LTPB6Z0V/p1695201699156729.

Mossaka commented 1 year ago

Thank you for conducting tests on the shims on multiple k8s distributions; your findings are invaluable for us maintainers to understand our operational landscape and potential challenges with each release. To automate this process, I suggest implementing integration tests via GitHub Actions within the upstream repositories, runwasi and containerd-wasm-shims. This would serve as a robust safeguard for future releases.

The challenge lies in, for one, writing tests for two repositories and, for two, accommodating the numerous Kubernetes distributions, as you listed a table. Despite these complexities, I'm willing to dedicate some time to the automation effort for both the runwasi and containerd-wasm-shims repositories.

Please note that I won't have much capacity until I come back from my vacation 🏝️

0xE282B0 commented 1 year ago

@Mossaka I don't expect you to do anything while you're on vacation, not even read github issues 😉.

0xE282B0 commented 1 year ago

AWS EKS

Prerequisites

Setup cluster

# Crate cluster
eksctl create cluster --version 1.27

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached..

❯ kubectl get pod  
NAME                             READY   STATUS    RESTARTS   AGE
lunatic-demo-785446bf5-sfhmw     2/2     Running   0          58s
wasm-slight-9566467b7-ntncd      2/2     Running   0          58s
wasm-spin-8cc64fd85-vvr97        2/2     Running   0          58s
wasm-wws-ff448f95d-gz7qk         2/2     Running   0          58s
wasmedge-demo-774b4489c5-z7blc   2/2     Running   0          59s
wasmer-demo-5b7c886887-pt7ds     2/2     Running   0          59s
wasmtime-demo-6878c5447-r46d8    2/2     Running   0          58s
0xE282B0 commented 1 year ago

CIVO K3s

Prerequisites

Setup cluster

# Crate cluster
civo kubernetes create kwasm
civo kubernetes config kwasm --save

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached..

❯ kubectl get pod                                                                                                                                             
NAME                             READY   STATUS    RESTARTS   AGE
wasm-slight-5646fd7cd5-q7mgl     2/2     Running   0          3m7s
wasmer-demo-558c75c49f-p9sq7     2/2     Running   0          3m7s
wasm-wws-c875b9664-s9b57         2/2     Running   0          3m7s
wasmedge-demo-7fb59fffdc-jqm2s   2/2     Running   0          3m7s
wasmtime-demo-6979d575-zs2hl     2/2     Running   0          3m7s
lunatic-demo-767757cf9-8r88m     2/2     Running   0          3m7s
wasm-spin-85bfb69795-cr6nv       2/2     Running   0          3m7s
0xE282B0 commented 1 year ago

Colima

Prerequisites

Setup cluster

# Crate cluster
colima start -k -r containerd --kubernetes-version v1.28.1+k3s1 --network-address

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached. Notes:

❯ kubectl get pod
NAME                             READY   STATUS    RESTARTS   AGE
wasm-slight-6467bcc5bc-dshjf     2/2     Running   0          2m3s
wasm-spin-74c4cf5c77-zc2pv       2/2     Running   0          2m3s
wasm-wws-888f6bc4b-lj8nh         2/2     Running   0          2m3s
lunatic-demo-6475554875-kv497    2/2     Running   0          2m3s
wasmedge-demo-5ff758d79-b92kt    2/2     Running   0          2m3s
wasmer-demo-857f947cb7-82kpn     2/2     Running   0          2m3s
wasmtime-demo-56c78ddd95-zmqhp   2/2     Running   0          2m3s
0xE282B0 commented 1 year ago

DigitalOcean Kubernetes

Prerequisites

Setup cluster

# Crate cluster
doctl kubernetes cluster create kwasm
doctl kubernetes cluster kubeconfig save kwasm      

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached.

❯ kubectl get pod
NAME                             READY   STATUS    RESTARTS   AGE
lunatic-demo-6475554875-fxrr4    2/2     Running   0          49s
wasm-slight-6467bcc5bc-jbkkg     2/2     Running   0          49s
wasm-spin-74c4cf5c77-22zkt       2/2     Running   0          49s
wasm-wws-888f6bc4b-xd8k5         2/2     Running   0          48s
wasmedge-demo-5ff758d79-r6h9m    2/2     Running   0          49s
wasmer-demo-857f947cb7-rsrzz     2/2     Running   0          49s
wasmtime-demo-56c78ddd95-j4cwb   2/2     Running   0          49s
0xE282B0 commented 1 year ago

Killercoda.com

Prerequisites

Setup cluster

visit: https://killercoda.com/playgrounds/scenario/kubernetes

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached.

NAME                             READY   STATUS    RESTARTS   AGE
lunatic-demo-785446bf5-zjg2w     2/2     Running   0          2m2s
wasm-slight-9566467b7-jcgjd      2/2     Running   0          2m2s
wasm-spin-8cc64fd85-zc5d2        2/2     Running   0          2m2s
wasm-wws-ff448f95d-lpx4n         2/2     Running   0          2m2s
wasmedge-demo-774b4489c5-bgdwf   2/2     Running   0          2m2s
wasmer-demo-5b7c886887-8lz7t     2/2     Running   0          2m2s
wasmtime-demo-6878c5447-t82gx    2/2     Running   0          2m2s
0xE282B0 commented 1 year ago

Minikube

Prerequisites

Setup cluster

# Crate cluster
minikube start --container-runtime='containerd'

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached.

NAME                             READY   STATUS    RESTARTS   AGE
lunatic-demo-785446bf5-8vdlc     2/2     Running   0          100s
wasm-slight-9566467b7-nchvc      2/2     Running   0          100s
wasm-spin-8cc64fd85-wwzc5        2/2     Running   0          100s
wasm-wws-ff448f95d-dcwhk         2/2     Running   0          99s
wasmedge-demo-774b4489c5-dcfn7   2/2     Running   0          100s
wasmer-demo-5b7c886887-j6qlv     2/2     Running   0          100s
wasmtime-demo-6878c5447-b9tj7    2/2     Running   0          100s
0xE282B0 commented 1 year ago

MicroK8s

Prerequisites

Setup cluster

# Crate cluster
# Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-1034-raspi aarch64)
sudo snap install microk8s --channel=1.28-strict/stable

# Install with 0.3.0-pre installer
helm repo add kwasm http://kwasm.sh/kwasm-operator/ --force-update
helm repo update
helm upgrade --install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator \
  --set kwasmOperator.autoProvision="true" \
  --set kwasmOperator.installerImage="ghcr.io/kwasm/kwasm-node-installer:pr-42"

Test execution

kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/runtimeclass.yaml
kubectl apply -f https://gist.githubusercontent.com/0xE282B0/1854ad6ed7ddaf4ad7d3032397056a38/raw/937c56555532d8f7dadedc9405c6764f3133073e/workload.yaml

Test results

Description: This was a smoke test, all containers start and have a sidecar container attached.

NAME                             READY   STATUS    RESTARTS   AGE
wasmedge-demo-5ff758d79-xj5fs    2/2     Running   0          2m9s
wasmer-demo-857f947cb7-nc99l     2/2     Running   0          2m9s
lunatic-demo-6475554875-lnc7b    2/2     Running   0          2m8s
wasm-slight-6467bcc5bc-k5l62     2/2     Running   0          2m7s
wasm-spin-74c4cf5c77-qtb9p       2/2     Running   0          2m8s
wasmtime-demo-56c78ddd95-8p9bc   2/2     Running   0          2m8s
wasm-wws-888f6bc4b-7cm27         2/2     Running   0          2m7s
0xE282B0 commented 1 year ago

For the v0.3.0 installer, the runwasi shims are built from the master (commit: ba5ab5ada5a4017622f6d93092345baf9cdfb507). With this test I'm confident to release the installer. When runwasi is released, I will create a patch release to use the same artifacts that are being released.

Thank you @jprendes and @Mossaka!