rajatjindal
commented
3 months ago Hi @byteshiva,
could you please re-run the command to create the cluster with --verbose flag to check what it is waiting for.
Open byteshiva opened 3 months ago
rajatjindal
commented
3 months ago Hi @byteshiva,
could you please re-run the command to create the cluster with --verbose flag to check what it is waiting for.
byteshiva
commented
3 months ago k3d cluster create wasm-cluster --image ghcr.io/spinkube/containerd-shim-spin/k3d:v0.13.1 --port "8081:80@loadbalancer"
Sure, here are the detailed logs after applying the --verbose flag:
```
k3d cluster create wasm-cluster --image ghcr.io/spinkube/containerd-shim-spin/k3d:v0.13.1 --port "8081:80@loadbalancer" --verbose
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:20.10.25 OSType:linux OS:NixOS 23.05 (Stoat) Arch:x86_64 CgroupVersion:2 CgroupDriver:systemd Filesystem:extfs InfoName:nixos}
DEBU[0000] Additional CLI Configuration:
cli:
api-port: ""
env: []
k3s-node-labels: []
k3sargs: []
ports:
- 8081:80@loadbalancer
registries:
create: ""
runtime-labels: []
runtime-ulimits: []
volumes: []
hostaliases: []
DEBU[0000] Configuration:
agents: 0
image: ghcr.io/spinkube/containerd-shim-spin/k3d:v0.13.1
network: ""
options:
k3d:
disableimagevolume: false
disableloadbalancer: false
disablerollback: false
loadbalancer:
configoverrides: []
timeout: 0s
wait: true
kubeconfig:
switchcurrentcontext: true
updatedefaultkubeconfig: true
runtime:
agentsmemory: ""
gpurequest: ""
hostpidmode: false
serversmemory: ""
registries:
config: ""
use: []
servers: 1
subnet: ""
token: ""
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha5} ObjectMeta:{Name:} Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:} Image:ghcr.io/spinkube/containerd-shim-spin/k3d:v0.13.1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[] Ulimits:[]}} Env:[] Registries:{Use:[] Create:
Error encountered while attempting to delete cluster using k3d
k3d cluster delete wasm-cluster --verbose
```
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:20.10.25 OSType:linux OS:NixOS 23.05 (Stoat) Arch:x86_64 CgroupVersion:2 CgroupDriver:systemd Filesystem:extfs InfoName:nixos}
DEBU[0000] Configuration:
{}
ERRO[0000] error getting loadbalancer config from k3d-wasm-cluster-serverlb: runtime failed to read loadbalancer config '/etc/confd/values.yaml' from node 'k3d-wasm-cluster-serverlb': Error response from daemon: Could not find the file /etc/confd/values.yaml in container 26f40673af81a72930dd7ba0a358bf2b35eabcfb4a571bdb50d9add39338fc14: file not found
INFO[0000] Deleting cluster 'wasm-cluster'
DEBU[0000] Cluster Details: &{Name:wasm-cluster Network:{Name:k3d-wasm-cluster ID: External:false IPAM:{IPPrefix:invalid Prefix IPsUsed:[] Managed:false} Members:[]} Token:DtZzZqxgaZUKhJeSfpDL Nodes:[0xc00001f6c0 0xc000334340] InitNode:
``` $ cat run.sh export NIXPKGS_ALLOW_UNFREE=1 nix-shell -E ' let nixpkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz") {}; in nixpkgs.mkShell { buildInputs = with nixpkgs; [ k3d kubectl kubernetes-helm docker ]; shellHook = "export KUBECONFIG=kubeconfig"; }' ```
rajatjindal
commented
3 months ago I just tried it on my laptop (Macbook M2 Pro), and it completed successfully.
k3d cluster create wasm-cluster --image ghcr.io/spinkube/containerd-shim-spin/k3d:v0.13.1 --port "8081:80@loadbalancer" --verbose
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:25.0.1 OSType:linux OS:Docker Desktop Arch:aarch64 CgroupVersion:2 CgroupDriver:cgroupfs Filesystem:extfs InfoName:docker-desktop}
--ommitted some logs for readability--
DEBU[0023] Setting new current-context 'k3d-wasm-cluster'
DEBU[0023] Wrote kubeconfig to 'kubeconfig'
INFO[0023] You can now use it like this:
kubectl cluster-info
[nix-shell:~/nix-spinkube]$I initiated a discussion on the K3d GitHub repository at https://github.com/k3d-io/k3d/discussions/1422.
I followed the steps below to set up a Spinkube cluster in a local environment using Talos Linux
But I see an error at:
kubectl logs -f simple-spinapp-56687588d9-jcpvt
Error from server (BadRequest): container "simple-spinapp" in pod "simple-spinapp-56687588d9-jcpvt" is waiting to start: ContainerCreating
Step 1:
```
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.3/cert-manager.yaml
kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.1.0/spin-operator.crds.yaml
kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.1.0/spin-operator.runtime-class.yaml
kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.1.0/spin-operator.shim-executor.yaml
helm install spin-operator \
--namespace spin-operator \
--create-namespace \
--version 0.1.0 \
--wait \
oci://ghcr.io/spinkube/charts/spin-operator
helm repo add kwasm http://kwasm.sh/kwasm-operator/
helm install \
kwasm-operator kwasm/kwasm-operator \
--namespace kwasm \
--create-namespace \
--set kwasmOperator.installerImage=ghcr.io/spinkube/containerd-shim-spin/node-installer:v0.13.1
kubectl annotate node --all kwasm.sh/kwasm-node=true
```
Step 2:
```
wget https://raw.githubusercontent.com/spinkube/spin-operator/main/config/samples/simple.yaml
kubectl apply -f simple.yaml
```
Step 3:
```
kubectl get pods,svc,nodes -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager pod/cert-manager-5b54fc556f-dd7fw 1/1 Running 0 12m
cert-manager pod/cert-manager-cainjector-7d8b6cf7b9-7sv66 1/1 Running 0 12m
cert-manager pod/cert-manager-webhook-7d4744b5ff-4vxts 1/1 Running 0 12m
default pod/simple-spinapp-56687588d9-jcpvt 0/1 ContainerCreating 0 7m1s
kube-system pod/coredns-85b955d87b-fmfjw 1/1 Running 0 14m
kube-system pod/coredns-85b955d87b-kz5h4 1/1 Running 0 14m
kube-system pod/kube-apiserver-talos-xwn-6bf 1/1 Running 0 14m
kube-system pod/kube-controller-manager-talos-xwn-6bf 1/1 Running 2 (15m ago) 13m
kube-system pod/kube-flannel-h95hg 1/1 Running 0 14m
kube-system pod/kube-flannel-lqv4w 1/1 Running 0 14m
kube-system pod/kube-proxy-ngzch 1/1 Running 0 14m
kube-system pod/kube-proxy-t7fh7 1/1 Running 0 14m
kube-system pod/kube-scheduler-talos-xwn-6bf 1/1 Running 2 (15m ago) 14m
kwasm pod/kwasm-operator-6c76c5f94b-cb5wt 1/1 Running 0 11m
spin-operator pod/spin-operator-controller-manager-565945c6f5-zhwx8 2/2 Running 0 12m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cert-manager service/cert-manager ClusterIP 10.107.105.237
reference:
byteshiva
commented
3 months ago I deployed a simple JavaScript Spin application on the GitHub Container Registry and made it public.
https://github.com/users/byteshiva/packages/container/package/hello-k3s
However, I'm encountering difficulties in creating the container, and I'm unsure of the reason behind it.
``` cat spinapp.yaml apiVersion: core.spinoperator.dev/v1alpha1 kind: SpinApp metadata: name: hello-k3s spec: image: "ghcr.io/byteshiva/hello-k3s:0.0.1" executor: containerd-shim-spin replicas: 2 ``` ``` kubectl apply -f spinapp.yaml ``` ``` [nix-shell:~/]$ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE cert-manager cert-manager-5b54fc556f-dd7fw 1/1 Running 0 109m cert-manager cert-manager-cainjector-7d8b6cf7b9-7sv66 1/1 Running 0 109m cert-manager cert-manager-webhook-7d4744b5ff-4vxts 1/1 Running 0 109m default hello-k3s-6f8f596bb9-6ctpz 0/1 ContainerCreating 0 6s default hello-k3s-6f8f596bb9-rb4g7 0/1 ContainerCreating 0 6s kube-system coredns-85b955d87b-fmfjw 1/1 Running 0 111m kube-system coredns-85b955d87b-kz5h4 1/1 Running 0 111m kube-system kube-apiserver-talos-xwn-6bf 1/1 Running 0 111m kube-system kube-controller-manager-talos-xwn-6bf 1/1 Running 2 (111m ago) 110m kube-system kube-flannel-h95hg 1/1 Running 0 111m kube-system kube-flannel-lqv4w 1/1 Running 0 111m kube-system kube-proxy-ngzch 1/1 Running 0 111m kube-system kube-proxy-t7fh7 1/1 Running 0 111m kube-system kube-scheduler-talos-xwn-6bf 1/1 Running 2 (112m ago) 110m kwasm kwasm-operator-6c76c5f94b-cb5wt 1/1 Running 0 107m spin-operator spin-operator-controller-manager-565945c6f5-zhwx8 2/2 Running 0 108m [nix-shell:~/]$ kubectl logs -f hello-k3s-6f8f596bb9-6ctpz Error from server (BadRequest): container "hello-k3s" in pod "hello-k3s-6f8f596bb9-6ctpz" is waiting to start: ContainerCreating ``` ``` kubectl describe hello-k3s-6f8f596bb9-6ctpz Warning FailedCreatePodSandBox 69s (x26 over 6m23s) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "spin" is configured ```
0xE282B0
commented
3 months ago Hi @byteshiva,
how does your RuntimeClass look?
byteshiva
commented
3 months ago Hi @byteshiva, how does your
RuntimeClasslook?
The RuntimeClass configuration looks like this
$cat spin-runtime-class.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: wasmtime-spin-v2
handler: spin
```
$ kubectl describe runtimeclass wasmtime-spin-v2
Name: wasmtime-spin-v2
Namespace:
Labels:
0xE282B0
commented
3 months ago How did you set up the K3s? Do you have access to the /var/lib/rancher/k3s/agent/etc/containerd/config.toml file?
byteshiva
commented
3 months ago How did you set up the K3s? Do you have access to the /var/lib/rancher/k3s/agent/etc/containerd/config.toml file?
Please note that I'm not using K3d/k3s. Instead, I have set it up on Talos Linux. You can set up Talos Linux using the guide at: https://www.talos.dev/v1.6/talos-guides/install/local-platforms/virtualbox/
**Controller Node Config** ``` $ talosctl cat /etc/containerd/config.toml -n "${CONTROLER_PLANE_IP}" version = 2 disabled_plugins = [ "io.containerd.grpc.v1.cri", "io.containerd.internal.v1.opt", "io.containerd.internal.v1.tracing", "io.containerd.nri.v1.nri", "io.containerd.snapshotter.v1.blockfile", "io.containerd.tracing.processor.v1.otlp", ] [debug] level = "info" format = "json" ``` **Agent/Worker Node Config** ``` $ talosctl cat /etc/containerd/config.toml -n "${WORKER_IP}" version = 2 disabled_plugins = [ "io.containerd.grpc.v1.cri", "io.containerd.internal.v1.opt", "io.containerd.internal.v1.tracing", "io.containerd.nri.v1.nri", "io.containerd.snapshotter.v1.blockfile", "io.containerd.tracing.processor.v1.otlp", ] [debug] level = "info" format = "json" ``` ``` $ talosctl services -n "${CONTROLER_PLANE_IP}" SERVICE STATE HEALTH LAST CHANGE LAST EVENT apid Running OK 18m55s ago Health check successful containerd Running OK 18m57s ago Health check successful cri Running OK 18m54s ago Health check successful dashboard Running ? 18m56s ago Process Process(["/sbin/dashboard"]) started with PID 1699 etcd Running OK 18m20s ago Health check successful kubelet Running OK 19m9s ago Health check successful machined Running OK 19m2s ago Health check successful trustd Running OK 20m9s ago Health check successful udevd Running OK 19m2s ago Health check successful ``` ``` $ talosctl ls /etc/ -n "${CONTROLER_PLANE_IP}" NAME . ca-certificates cni containerd cri extensions.yaml hosts kubernetes localtime lvm machine-id nfsmount.conf os-release pki resolv.conf ssl ```
squillace
commented
3 months ago I like Talos! But realistically we would not have an "instead" sample; no harm in having several. :-)
0xE282B0
commented
3 months ago Please note that I'm not using K3d/k3s. Instead, I have set it up on Talos Linux.
In that case the error message is correct, the containerd shim is not installed.
With KWasm or the runtime-class-manager the spin shim can be installed on a variety of Kubernetes distributions. But that is not working with Talos. But we can build a talos-extensions that adds the shim.
Only Talos on Docker can't execute extensions afaik. Are you using Talos on Docker?
byteshiva
commented
3 months ago Only Talos on Docker can't execute extensions afaik. Are you using Talos on Docker?
No, I'm not using Talos on Docker. I'm currently running Talos on VirtualBox using an ISO - metal-amd64.iso
0xE282B0
commented
3 months ago Great! @saiyam1814 already added an extension for WasmEdge, so we just need to do the same for Spin 👍
0xE282B0
commented
3 months ago Hi @byteshiva
I was not able to verify it today, but this should work:
https://github.com/siderolabs/extensions/compare/main...0xE282B0:talos-extensions:feat-spin-extension
I'll open a PR when I made sure it runs with a real Talos.
0xE282B0
commented
3 months ago Hi @byteshiva, I opened a PR for the official Talos extensions repo: https://github.com/siderolabs/extensions/pull/355
I tested it on Digital Ocean and a Raspberry Pi 4. Steps to reproduce:
git clone https://github.com/0xE282B0/talos-extensions.git -b feat-spin-extensioncd talos-extensions && make REGISTRY=docker.io USERNAME=0xe282b0 TARGETS="spin" PUSH=true
If you get the error message: "Makefile:106: *** missing separator. Stop." aou need to delete the lines 99-147 from the Makefile.docker run --rm -t -v $PWD/_out:/out -v /dev:/dev \
--privileged ghcr.io/siderolabs/imager:v1.6.7 digital-ocean \
--arch amd64 \
--system-extension-image docker.io/0xe282b0/spin:v0.13.1talosctl apply-config --insecure --mode=interactive --nodes <PRIVATE_IP>docker run --rm -t -v $PWD/_out:/out -v /dev:/dev \
--privileged ghcr.io/siderolabs/imager:v1.6.7 metal \
--arch arm64 \
--system-extension-image docker.io/0xe282b0/spin:v0.13.1 \
--board rpi_genericUse the out/metal-rpi_generic-arm64.raw.xz to flash a sdcard e.g. with Raspberry Pi Imager
❯ talosctl --nodes 192.168.188.58 get extensions
NODE NAMESPACE TYPE ID VERSION NAME VERSION
192.168.188.58 runtime ExtensionStatus 0 1 spin v0.13.1apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: wasmtime-spin-v2
handler: spinapiVersion: v1
kind: Pod
metadata:
name: spin-test
spec:
containers:
- command:
- /
image: ghcr.io/spinkube/containerd-shim-spin/examples/spin-rust-hello
name: spin-test
runtimeClassName: wasmtime-spin-v2kubectl get pod
NAME READY STATUS RESTARTS AGE
spin-test 1/1 Running 0 56m# Finally test the hello spin app 🥳
kubectl port-forward pod/spin-test 8000:80
curl localhost:8000/hello
> Hello world from Spin!Now everything is set up to use SpinKube! 🚀
@tpmccallum is SpinKube on Talos Linux something we should add to the documentation?
saiyam1814
commented
3 months ago Nice! @0xE282B0 let me know if any help needed!!
Description: I would like to request the use of Talos Linux instead of k3d for the deployment. Currently, I am facing issues when trying to pull k3d images. Here are the details of the problem:
The below deployment and services to work in talos linux.
I followed the instruction as provided in https://www.spinkube.dev/docs/spin-operator/installation/installing-with-helm/.
Please assist in resolving this issue and providing guidance on how to use Talos Linux for the deployment.
ps: I plan to use Talos Linux instead of K3d. https://www.spinkube.dev/docs/spin-operator/quickstart/
Error: The k3d example fails to work in my NixOS setup. It's stuck at the below