voigt
commented
3 months ago The install-pods of kwasm do not terminate with status Unknown, but Completed. The main difference is, that kwasms install script uses the system schedulers restart functionality.
In case of systemd this means, that containerd receives a SIGTERM and only after 90 seconds a SIGKILL (source).
node-installer directly sends a SIGHUP to the containerd process, which seems to me to be the issue.
As part of #68 I investigated an issue in the containerd restart routine. When the node-installer installs a runtime and restarts containerd, the corresponding pod terminates with status
UnknownOverview:
Logs of Pod with status
UnknownLogs of Pod with status
CompletedThe
Completedpod only gets scheduled in the first place, as the first one did not terminated successfully; even though the actual job (rewriting containerd config and removing the binary) is done. As a result, the second run of the job has nothing left todo.Description of Pod with Status
Unknown
```bash Name: kwasm-worker-spin-v2-install-n82d9 Namespace: default Priority: 0 Service Account: default Node: kwasm-worker/192.168.228.5 Start Time: Mon, 20 May 2024 22:49:35 +0200 Labels: batch.kubernetes.io/controller-uid=7878f58f-1b99-4e81-99f1-7bd5b7bf54ac batch.kubernetes.io/job-name=kwasm-worker-spin-v2-install controller-uid=7878f58f-1b99-4e81-99f1-7bd5b7bf54ac job-name=kwasm-worker-spin-v2-install Annotations:kubectl describe po kwasm-worker-spin-v2-install-n82d9Entire resource of Job (e.g. for recreation of the bug)
```bash apiVersion: batch/v1 kind: Job metadata: annotations: kwasm.sh/nodeName: kwasm-worker kwasm.sh/operation: install kwasm.sh/shimName: spin-v2 labels: kwasm-worker-spin-v2-install: "true" kwasm.sh/job: "true" kwasm.sh/operation: install kwasm.sh/shimName: spin-v2 name: kwasm-worker-spin-v2-install namespace: default spec: backoffLimit: 6 completionMode: NonIndexed completions: 1 manualSelector: false parallelism: 1 podReplacementPolicy: TerminatingOrFailed selector: matchLabels: batch.kubernetes.io/controller-uid: 7878f58f-1b99-4e81-99f1-7bd5b7bf54ac suspend: false template: metadata: creationTimestamp: null labels: batch.kubernetes.io/controller-uid: 7878f58f-1b99-4e81-99f1-7bd5b7bf54ac batch.kubernetes.io/job-name: kwasm-worker-spin-v2-install controller-uid: 7878f58f-1b99-4e81-99f1-7bd5b7bf54ac job-name: kwasm-worker-spin-v2-install spec: containers: - args: - install - -H - /mnt/node-root - -r - spin-v2 env: - name: HOST_ROOT value: /mnt/node-root - name: SHIM_FETCH_STRATEGY value: /mnt/node-root image: ghcr.io/spinkube/node-installer:latest-feat-add_shim_downloader imagePullPolicy: IfNotPresent name: provisioner resources: {} securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/node-root name: root-mount - mountPath: /assets name: shim-download dnsPolicy: ClusterFirst hostPID: true initContainers: - env: - name: SHIM_NAME value: spin-v2 - name: SHIM_LOCATION value: https://github.com/spinkube/containerd-shim-spin/releases/download/v0.14.1/containerd-shim-spin-v2-linux-aarch64.tar.gz image: ghcr.io/spinkube/shim-downloader:latest-feat-add_shim_downloader imagePullPolicy: IfNotPresent name: downloader resources: {} securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /assets name: shim-download nodeName: kwasm-worker restartPolicy: Never schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - emptyDir: {} name: shim-download - hostPath: path: / type: "" name: root-mount status: completionTime: "2024-05-20T20:50:03Z" conditions: - lastProbeTime: "2024-05-20T20:50:03Z" lastTransitionTime: "2024-05-20T20:50:03Z" status: "True" type: Complete failed: 1 ready: 0 startTime: "2024-05-20T20:49:35Z" succeeded: 1 terminating: 0 uncountedTerminatedPods: {} ```While the goal of installing/uninstalling the shim is achieved, this is not a desired behavior and desires for a solution.