SSL Error on outgoing request

[stevesloka]

I have a Spin app running in k8s that works fine until I make an outgoing request to another resource. This using the latest 0.9.2.

When do I get a generic error from the spin's stdout:

Handler returned an error: Uncaught SyntaxError: unexpected token: 'Uncaught'                                                                                                                                        
    at <input>:1                                                                                                                                                                                                     
    at parse (native)                                                                                                                                                                                                
    at <anonymous> (script.js:397)                                                                                                                                                                                   
    at call (native)                                                                                                                                                                                                 
    at step (script.js:306)                                                                                                                                                                                          
    at <anonymous> (script.js)                                                                                                                                                                                       
    at fulfilled (script.js)   

But looking at the containerd logs, I see this more specific error: Outbound HTTP connection error: URL https://dummyjson.com/products/1, error detail reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("dummyjson.com")), port: None, path: "/products/1", query: None, fragment: None }, source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: Some(Ssl(ErrorStack([Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1889 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" })) }

Any quick work arounds that I can look at to resolve?

[Mossaka]

Could this help? https://github.com/deislabs/containerd-wasm-shims/blob/main/images/slight/Dockerfile#L10C1-L10C36

[stevesloka]

I can try that @Mossaka, I didn't have to before with other versions of the shims.

[Mossaka]

Okay if you didn't run into this before with other versions, it might be a regression from either the Spin runtime or containerd-shim-wasm crate.

@jprendes have you seen this issue before?

[jprendes]

I haven't seen it before.

@stevesloka could you share a reproduction example?

[stevesloka]

@jprendes yup, I essentially have an OCI image that I build and add the compiled wasm into. This all runs in k8s on EKS. I just was working to support Spin v1.5 so I used the latest 0.9.2 containerd-wasm-shim and deployed that into my cluster.

Doing some testing I noticed the runtime errors, locally they all work fine (i.e. not using the shim).

Here's my simple Spin app:

const encoder = new TextEncoder("utf-8")
const decoder = new TextDecoder("utf-8")

export async function handleRequest(request) {
    const res = await fetch("https://dummyjson.com/products/1")

    const body = decoder.decode(await res.arrayBuffer() || new Uint8Array())

    return {
        status: 200,
        body: encoder.encode(body).buffer
    }
}

If I have code that just returns static data, say a string, it works fine, but whenever I make an outgoing call as in the example previous, I get the containerd error noted in the first post.

Are there any other details that I can share that would help out?

[hrak]

I ran into exactly the same issue while experimenting with Spin and k8s during a hackathon. Eventually found out by digging through containerd logs of the workers. I tried both the Go and Python version of the outbound http example. The latter gave me a non-descriptive AssertionError: runtime-error (error 4) in the browser.

Dockerfile used:

FROM scratch
COPY ./spin.toml /spin.toml
COPY ./app.wasm /app.wasm
ENTRYPOINT ["/spin.toml"]

manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasm-random-animal-fact
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wasm-random-animal-fact
  template:
    metadata:
      labels:
        app: wasm-random-animal-fact
    spec:
      runtimeClassName: spin-test
      containers:
        - name: wasm-random-animal-fact
          imagePullPolicy: Always
          image: docker.io/hrak/random-animal-fact-wasm:0.0.2
          command: ["/"]
---
apiVersion: v1
kind: Service
metadata:
  name: wasm-random-animal-fact
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: wasm-random-animal-fact

Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]: time="2023-12-01T12:32:12.282773715Z" level=info msg="instantiate_async;"
Dec 01 12:31:25 testor-md-0-w5sgd containerd[1405]: time="2023-12-01T12:31:25.994484003Z" level=warn msg="Outbound HTTP connection error: URL https://random-data-api.fermyon.app/animals/json, error detail reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("random-data-api.fermyon.app")), port: None, path: "/animals/json", query: None, fragment: None }, source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: Some(Ssl(ErrorStack([Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1889 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" })) }"ec 01 12:32:12 testor-md-0-w5sgd containerd[1405]: time="2023-12-01T12:32:12.482456071Z" level=error msg="Error processing request: error while executing at wasm backtrace:
Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]:     0: 0xcac9 - <unknown>!runtime.runtimePanicAt.llvm.2870195029891959485
Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]:     1: 0x59c1 - <unknown>!runtime.nilPanic.llvm.2870195029891959485
Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]:     2: 0x35958 - <unknown>!spin_http_handle_http_request
Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]:     3: 0x360af - <unknown>!__wasm_export_spin_http_handle_http_request
Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]:     4: 0x470ae - wit-component:adapter:wasi_snapshot_preview1!fermyon:spin/inbound-http#handle-request
Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]: Caused by:
Dec 01 12:32:12 testor-md-0-w5sgd containerd[1405]:     wasm trap: wasm `unreachable` instruction executed"

My eventual solution was this:

FROM --platform=${BUILDPLATFORM} gcr.io/distroless/static AS certs
FROM scratch
COPY --from=certs /etc/ssl /usr/local/ssl
COPY ./spin.toml /spin.toml
COPY ./app.wasm /app.wasm
ENTRYPOINT ["/spin.toml"]

[radu-matei]

This is now fixed in https://github.com/spinkube/containerd-shim-spin/issues/44.