Inconsistent behavior with Wasm enabled

divya-mohan0209 commented 6 months ago

Actual Behavior

Context:

I tried out the SpinKube x Rancher Desktop integration detailed on this page. It works seamlessly for the hello-world application detailed there & on the Fermyon blog.

However, when I tried installing some of the other complex templates and containerizing them, such as

or even templates of my own

Rancher Docs template

there is inconsistent behaviour, i.e. they sometimes work and most of the time, they don't.

This is when the spin applications themselves work fine on my machine.

Steps to Reproduce

This is a repo with the template I created that has the steps listed.

The app, itself, works locally. But when containerizing it using SpinKube, it doesn't work.

I've also tried other more complex (and official) templates listed on the Fermyon website like

but they don't work.

Result

The pods enter the CrashLoopBackOff stage and are terminated with the following message: Last state: Terminated with 137: Error.

Expected Behavior

The containerized apps should work and the pods should come up. However, this behavior is wildly inconsistent.

Additional Information

Last state: Terminated with 137: Error - Since this error typically points to an error with memory. I tried increasing the memory assigned to pods and it didn't help.

I tried resetting the Kubernetes cluster and restoring Rancher Desktop to its factory settings (individually, of course). None of those approaches helped and in fact, had the opposite effect. If the templates were working before the factory reset or the cluster reset, they stopped working after. (Of course, I shouldn't have tried to fix what wasn't broken by resetting it 😆 But I did it anyway for the sake of reproducibility)

Lastly, I wasn't sure where the error was, so I filed it against this repo. I'll also open an issue against the SpinKube GitHub repo.

Rancher Desktop Version

1.13.1

Rancher Desktop K8s Version

1.29.3, 1.28.9

Which container engine are you using?

containerd (nerdctl)

What operating system are you using?

macOS

Operating System / Build Version

Sonoma 14.1

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

No response

divya-mohan0209 commented 6 months ago

Issue opened against the SpinKube repo

saiyam1814 commented 6 months ago

Another example not working

clone this https://github.com/fermyon/ai-examples/tree/main/sentiment-analysis-ts

do

spin registry push ttl.sh/demosenti:1h

Create certificate curl -sfL https://curl.se/ca/cacert.pem -o ca-certificates.crt && kubectl create secret generic spin-ca --from-file=ca-certificates.crt

Add spinapp.yaml

apiVersion: core.spinoperator.dev/v1alpha1
kind: SpinApp
metadata:
  name: spin-sentiment-analysis
spec:
  image: "ttl.sh/demosenti:1h"
  executor: containerd-shim-spin
  replicas: 2
  runtimeConfig:
    llmCompute:
      type: "remote_http"
      options:
        - name : "url"
          value: "https://fermyon-cloud-gpu-uguyedt7.fermyon.app"
        - name: "auth_token"
          value: "addtoken"
  volumes:
    - name: ca-certs
      secret:
        secretName: spin-ca
        items:
          - key: ca-certificates.crt
            path: ca-certificates.crt
  volumeMounts:
    - name: ca-certs
      mountPath: "/etc/ssl/certs/ca-certificates.crt"
      readOnly: true
      subPath: ca-certificates.crt

jandubois commented 6 months ago

I can reproduce this failure. I believe it is due to Rancher Desktop 1.13 shipping with containerd-shim-spin-v2 version 0.11.1 (from https://github.com/deislabs/containerd-wasm-shims/releases/tag/v0.11.1).

I've tested with the 0.14.1 shim (from https://github.com/spinkube/containerd-shim-spin/releases/tag/v0.14.1) and the repro case deploys successfully.

I suspect the problem was due to a runwasi app size limit that seems to have been lifted in the 0.13.0 release of the shim:

It also pulls in updates in runwasi to https://github.com/containerd/runwasi/pull/507, enabling support for larger Spin apps (greater than 16 MB).

You can download the latest shim and put it into the containerd-shims directory, e.g. on macOS:

$ ls ~/Library/Application\ Support/rancher-desktop/containerd-shims
containerd-shim-spin-v2

After installing the shim you need to restart Rancher Desktop. You can verify that the new shim is being used via:

$ rdctl shell containerd-shim-spin-v2 -v
containerd-shim-spin-v2:
  Runtime: spin
  Version: 0.14.1
  Revision: eed8342b090fc8b

Rancher Desktop 1.14.0 will ship with spin 2.5.0 and containerd-shim-spin-v2 0.14.1 (or later).

Feel free to reopen this issue if you still have problems after upgrading the shim!

rancher-sandbox / rancher-desktop