Open voigt opened 3 months ago
The install-pods of kwasm do not terminate with status Unknown
, but Completed
. The main difference is, that kwasms install script uses the system schedulers restart
functionality.
In case of systemd this means, that containerd receives a SIGTERM
and only after 90 seconds a SIGKILL
(source).
node-installer
directly sends a SIGHUP
to the containerd process, which seems to me to be the issue.
As part of #68 I investigated an issue in the containerd restart routine. When the node-installer installs a runtime and restarts containerd, the corresponding pod terminates with status
Unknown
Overview:
Logs of Pod with status
Unknown
Logs of Pod with status
Completed
The
Completed
pod only gets scheduled in the first place, as the first one did not terminated successfully; even though the actual job (rewriting containerd config and removing the binary) is done. As a result, the second run of the job has nothing left todo.Description of Pod with Status
Unknown
```bash Name: kwasm-worker-spin-v2-install-n82d9 Namespace: default Priority: 0 Service Account: default Node: kwasm-worker/192.168.228.5 Start Time: Mon, 20 May 2024 22:49:35 +0200 Labels: batch.kubernetes.io/controller-uid=7878f58f-1b99-4e81-99f1-7bd5b7bf54ac batch.kubernetes.io/job-name=kwasm-worker-spin-v2-install controller-uid=7878f58f-1b99-4e81-99f1-7bd5b7bf54ac job-name=kwasm-worker-spin-v2-install Annotations:kubectl describe po kwasm-worker-spin-v2-install-n82d9
Entire resource of Job (e.g. for recreation of the bug)
```bash apiVersion: batch/v1 kind: Job metadata: annotations: kwasm.sh/nodeName: kwasm-worker kwasm.sh/operation: install kwasm.sh/shimName: spin-v2 labels: kwasm-worker-spin-v2-install: "true" kwasm.sh/job: "true" kwasm.sh/operation: install kwasm.sh/shimName: spin-v2 name: kwasm-worker-spin-v2-install namespace: default spec: backoffLimit: 6 completionMode: NonIndexed completions: 1 manualSelector: false parallelism: 1 podReplacementPolicy: TerminatingOrFailed selector: matchLabels: batch.kubernetes.io/controller-uid: 7878f58f-1b99-4e81-99f1-7bd5b7bf54ac suspend: false template: metadata: creationTimestamp: null labels: batch.kubernetes.io/controller-uid: 7878f58f-1b99-4e81-99f1-7bd5b7bf54ac batch.kubernetes.io/job-name: kwasm-worker-spin-v2-install controller-uid: 7878f58f-1b99-4e81-99f1-7bd5b7bf54ac job-name: kwasm-worker-spin-v2-install spec: containers: - args: - install - -H - /mnt/node-root - -r - spin-v2 env: - name: HOST_ROOT value: /mnt/node-root - name: SHIM_FETCH_STRATEGY value: /mnt/node-root image: ghcr.io/spinkube/node-installer:latest-feat-add_shim_downloader imagePullPolicy: IfNotPresent name: provisioner resources: {} securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/node-root name: root-mount - mountPath: /assets name: shim-download dnsPolicy: ClusterFirst hostPID: true initContainers: - env: - name: SHIM_NAME value: spin-v2 - name: SHIM_LOCATION value: https://github.com/spinkube/containerd-shim-spin/releases/download/v0.14.1/containerd-shim-spin-v2-linux-aarch64.tar.gz image: ghcr.io/spinkube/shim-downloader:latest-feat-add_shim_downloader imagePullPolicy: IfNotPresent name: downloader resources: {} securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /assets name: shim-download nodeName: kwasm-worker restartPolicy: Never schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - emptyDir: {} name: shim-download - hostPath: path: / type: "" name: root-mount status: completionTime: "2024-05-20T20:50:03Z" conditions: - lastProbeTime: "2024-05-20T20:50:03Z" lastTransitionTime: "2024-05-20T20:50:03Z" status: "True" type: Complete failed: 1 ready: 0 startTime: "2024-05-20T20:49:35Z" succeeded: 1 terminating: 0 uncountedTerminatedPods: {} ```While the goal of installing/uninstalling the shim is achieved, this is not a desired behavior and desires for a solution.