Update SpinKube to wait for cert-manager

[mikkelhegn]

Thank you for wanting to submit a Pull Request to the Civo Kubernetes Marketplace repository!

If your pull request is to submit a new application to the marketplace, please answer the following questions:

• [ ] Have you followed the guidelines in our Contributing document in setting up the application?
• [ ] Have you checked that the application your submission is proposing to add does not yet exist on the Marketplace, or as a pull request to be processed?
• [ ] Have you tested your application on the Civo managed Kubernetes service? Please include a screenshot.
• [ ] Can you confirm that the software you are submitting licensed for use in a publicly-available Kubernetes marketplace?
• [ ] Have you included the specific version of the application to be installed?

If your pull request concerns an existing Marketplace application, please make sure you have:

• [x] Notified the Maintainer of the application in this pull request so that they are aware of your proposal.
• [x] Outlined the changes you are proposing, and the reasons these are required (e.g. stability, compatibility with new versions of Kubernetes implementations, etc).
• [x] Tested that the application works with the proposed updates applied (including a screenshot).
• [ ] Updated the version number of the app if that has changed.

@saiyam1814 - This is an update to the logic to wait for cert-manager installation.

[saiyam1814]

@mikkelhegn the existing check should wait for 300 seconds, did you experience even more time taken thatn that for cert manager to be installed?

[mikkelhegn]

My observation is that the command fails because the namespace -n cert-manager cannot be found when it's run, so it just errors our and continues.

[saiyam1814]

One quick question on which Kubernetes cluster you tried? Tlos ro K3s?

[mikkelhegn]

I use k3s: civo kubernetes create -a spinkube --nodes 2

[saiyam1814]

Currently Spinkube only works on Talos as the shim is present there. For it to work with K3s, shim has to be installed separately

# Add Helm repository if not already done
helm repo add kwasm http://kwasm.sh/kwasm-operator/

# Install KWasm operator
helm install \
  kwasm-operator kwasm/kwasm-operator \
  --namespace kwasm \
  --create-namespace \
  --set kwasmOperator.installerImage=ghcr.io/spinkube/containerd-shim-spin/node-installer:v0.13.1

# Provision Nodes
kubectl annotate node --all kwasm.sh/kwasm-node=true

This has to be done first and then the app will work for k3s .

[mikkelhegn]

Ok, that makes sense. However, I don't think that change sthe logic needed for the install to wait. Let me test it out. Would also be great if the app checked for the distribution, and failed if k3s was identified.

[saiyam1814]

Agree, there has been in internal discussion for the aps to have cluster support field in manifest where we can sepcify it its for one or both and then display accordingly. no timelines yet. Meanwhile we can add custom logic in the intall script to check if its k3s then add that kwasm thind otherwise run the other part.

[mikkelhegn]

Ok - I'll focus this PR on identification of Talos and the install logic. Will not focus on k3s support for now.

[mikkelhegn]

Here's the error I see when running the app:

{"time":"2024-04-22T07:55:04.753259663Z","level":"INFO","msg":"Cloneing git repo https://github.com/civo/kubernetes-marketplace\n"}
{"time":"2024-04-22T07:55:04.753452601Z","level":"INFO","msg":"Creating temp dir to clone git repo"}
{"time":"2024-04-22T07:55:04.753570928Z","level":"INFO","msg":"Created temp dir: /tmp/prefix4120809582"}
{"time":"2024-04-22T07:55:07.905134059Z","level":"INFO","msg":"Validating that app exists: spinkube\n"}
{"time":"2024-04-22T07:55:07.905299658Z","level":"INFO","msg":"Running App Install"}
{"time":"2024-04-22T07:55:17.399627543Z","level":"INFO","msg":"Cheking the install.sh is preset or not"}
{"time":"2024-04-22T07:55:17.399811944Z","level":"INFO","msg":"Running the install.sh"}
{"time":"2024-04-22T07:55:17.399986028Z","level":"INFO","msg":"Making the shell file executable"}
{"time":"2024-04-22T07:55:17.402535219Z","level":"INFO","msg":"Command output","stdout":""}
{"time":"2024-04-22T07:55:17.402586623Z","level":"ERROR","msg":"Command output","stderr":""}
{"time":"2024-04-22T07:55:17.40259886Z","level":"INFO","msg":"Running the shell file"}
{"time":"2024-04-22T07:55:17.402707384Z","level":"INFO","msg":"Shell file contents","shellFile":"IyEvYmluL3NoCiNDZXJ0IG1hbmFnZXIgY2hlY2sgCgprdWJlY3RsIHdhaXQgLS1mb3I9Y29uZGl0aW9uPWF2YWlsYWJsZSAtLXRpbWVvdXQ9MzAwcyBkZXBsb3ltZW50L2NlcnQtbWFuYWdlci13ZWJob29rICAtbiBjZXJ0LW1hbmFnZXIKCmt1YmVjdGwgYXBwbHkgLWYgaHR0cHM6Ly9naXRodWIuY29tL3NwaW5rdWJlL3NwaW4tb3BlcmF0b3IvcmVsZWFzZXMvZG93bmxvYWQvdjAuMS4wL3NwaW4tb3BlcmF0b3IucnVudGltZS1jbGFzcy55YW1sCmt1YmVjdGwgYXBwbHkgLWYgaHR0cHM6Ly9naXRodWIuY29tL3NwaW5rdWJlL3NwaW4tb3BlcmF0b3IvcmVsZWFzZXMvZG93bmxvYWQvdjAuMS4wL3NwaW4tb3BlcmF0b3IuY3Jkcy55YW1sCmt1YmVjdGwgYXBwbHkgLWYgaHR0cHM6Ly9naXRodWIuY29tL3NwaW5rdWJlL3NwaW4tb3BlcmF0b3IvcmVsZWFzZXMvZG93bmxvYWQvdjAuMS4wL3NwaW4tb3BlcmF0b3Iuc2hpbS1leGVjdXRvci55YW1sCmhlbG0gaW5zdGFsbCBzcGluLW9wZXJhdG9yIFwKICAtLW5hbWVzcGFjZSBzcGluLW9wZXJhdG9yIFwKICAtLWNyZWF0ZS1uYW1lc3BhY2UgXAogIC0tdmVyc2lvbiAwLjEuMCBcCiAgLS13YWl0IFwKICBvY2k6Ly9naGNyLmlvL3NwaW5rdWJlL2NoYXJ0cy9zcGluLW9wZXJhdG9yCg=="}
Error: exit status 1
Usage:
  marketplace-installer install [flags]

Examples:
install <app name>

Flags:
  -h, --help   help for install

Global Flags:
  -d, --git-url string   The git repo to clone from (default "https://git.civo.com/civo/marketplace.git")

{"time":"2024-04-22T07:55:24.579238977Z","level":"INFO","msg":"Command output","stdout":"runtimeclass.node.k8s.io/wasmtime-spin-v2 created\ncustomresourcedefinition.apiextensions.k8s.io/spinappexecutors.core.spinoperator.dev created\ncustomresourcedefinition.apiextensions.k8s.io/spinapps.core.spinoperator.dev created\nspinappexecutor.core.spinoperator.dev/containerd-shim-spin created\n"}
{"time":"2024-04-22T07:55:24.579272602Z","level":"ERROR","msg":"Command output","stderr":"Error from server (NotFound): namespaces \"cert-manager\" not found\nPulled: ghcr.io/spinkube/charts/spin-operator:0.1.0\nDigest: sha256:919e2a3b513eeba58a16581f65cfe86194bf5d6a1ae37ef07e3e0aee7d1e52be\nW0422 07:55:24.409852      40 warnings.go:70] would violate PodSecurity \"restricted:latest\": seccompProfile (pod or containers \"manager\", \"kube-rbac-proxy\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")\nError: INSTALLATION FAILED: 2 errors occurred:\n\t* Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": dial tcp 10.109.204.8:443: connect: connection refused\n\t* Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": dial tcp 10.109.204.8:443: connect: connection refused\n\n\n"}

[mikkelhegn]

For identification of Talos vs. k3s - do you think this is viable? (e.g., look for Thalos in the osImage name)?

if $(kubectl get nodes -o json | jq .items.[].status.nodeInfo.osImage | grep -q "Thalos"); then Thalos=true; else Thalos=false; fi

# Thalos cluster
> k get nodes -o json | jq .items.[].status.nodeInfo.osImage
"Talos (v1.5.0-civo)"
"Talos (v1.5.0-civo)"

# K3s cluster
> k get nodes -o json | jq .items.[].status.nodeInfo.osImage
"Alpine Linux v3.18"
"Alpine Linux v3.18"

[saiyam1814]

I think if we take the version, It might change in future with different Kubernetes version. I think that Talos should remain. So we cn just have a single check if image name has Talos then do as is else we add that kwas thing as well.

[mikkelhegn]

What do you think about bumping the version to 0.2.0? It's part of the PR now.

I also added a note to the post_install.md file about only Talos being supported, but I'm not sure if that's the right place for that? Please advise.