failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount

[tekumara]

Error: failed to generate container "73c7a50781eaf5de74ab1f95568a7bc7e26016fd4a06aa28bf8ea2a79be3f9dd" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount

Expand below to see more.

kubectl describe pods -n kube-image builder-btls

``` Name: builder-btlsg Namespace: kube-image Priority: 0 Node: k3d-kim-server-0/172.22.0.2 Start Time: Sun, 27 Mar 2022 10:49:58 +1100 Labels: app=kim app.kubernetes.io/component=builder app.kubernetes.io/managed-by=kim app.kubernetes.io/name=kim component=builder controller-revision-hash=7bb6779b98 pod-template-generation=1 Annotations: Status: Pending IP: 172.22.0.2 IPs: IP: 172.22.0.2 Controlled By: DaemonSet/builder Init Containers: rshared-tmp: Container ID: containerd://949bd0c0307b7e9bd307fe6fdc154baac68c2807843aef74914294af5c622087 Image: docker.io/moby/buildkit:v0.8.3 Image ID: docker.io/moby/buildkit@sha256:171689e43026533b48701ab6566b72659dd1839488d715c73ef3fe387fab9a80 Port: Host Port: Command: sh -c Args: (if mountpoint $_DIR; then set -x; nsenter -m -p -t 1 -- env PATH=$_PATH sh -c 'mount --make-rshared $_DIR'; fi) || true State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 27 Mar 2022 10:50:16 +1100 Finished: Sun, 27 Mar 2022 10:50:16 +1100 Ready: True Restart Count: 0 Environment: _DIR: /tmp _PATH: /usr/sbin:/usr/bin:/sbin:/bin:/bin/aux Mounts: /tmp from host-tmp (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro) rshared-buildkit: Container ID: containerd://d61fd8440158f9e282cbcd0fcf77fa2c24e1e5826c3644e97b7bbe6cf82eb944 Image: docker.io/moby/buildkit:v0.8.3 Image ID: docker.io/moby/buildkit@sha256:171689e43026533b48701ab6566b72659dd1839488d715c73ef3fe387fab9a80 Port: Host Port: Command: sh -c Args: (if mountpoint $_DIR; then set -x; nsenter -m -p -t 1 -- env PATH=$_PATH sh -c 'mount --make-rshared $_DIR'; fi) || true State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 27 Mar 2022 10:50:16 +1100 Finished: Sun, 27 Mar 2022 10:50:16 +1100 Ready: True Restart Count: 0 Environment: _DIR: /var/lib/buildkit _PATH: /usr/sbin:/usr/bin:/sbin:/bin:/bin/aux Mounts: /var/lib/buildkit from host-var-lib-buildkit (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro) rshared-containerd: Container ID: containerd://49825e4b0231bfbd98f166208902e31a07c80f47ee2b754a17f3c9cdcef93a5c Image: docker.io/moby/buildkit:v0.8.3 Image ID: docker.io/moby/buildkit@sha256:171689e43026533b48701ab6566b72659dd1839488d715c73ef3fe387fab9a80 Port: Host Port: Command: sh -c Args: (if mountpoint $_DIR; then set -x; nsenter -m -p -t 1 -- env PATH=$_PATH sh -c 'mount --make-rshared $_DIR'; fi) || true State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 27 Mar 2022 10:50:17 +1100 Finished: Sun, 27 Mar 2022 10:50:17 +1100 Ready: True Restart Count: 0 Environment: _DIR: /var/lib/rancher _PATH: /usr/sbin:/usr/bin:/sbin:/bin:/bin/aux Mounts: /var/lib/rancher from host-containerd (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro) Containers: buildkit: Container ID: Image: docker.io/moby/buildkit:v0.8.3 Image ID: Port: 1234/TCP Host Port: 1234/TCP Args: --addr=unix:///run/buildkit/buildkitd.sock --addr=tcp://0.0.0.0:1234 --containerd-worker=true --containerd-worker-addr=/run/k3s/containerd/containerd.sock --containerd-worker-gc --oci-worker=false --tlscacert=/certs/ca/tls.crt --tlscert=/certs/server/tls.crt --tlskey=/certs/server/tls.key State: Waiting Reason: CreateContainerError Ready: False Restart Count: 0 Liveness: exec [buildctl debug workers] delay=5s timeout=1s period=20s #success=1 #failure=3 Readiness: exec [buildctl debug workers] delay=5s timeout=1s period=20s #success=1 #failure=3 Environment: Mounts: /certs/ca from certs-ca (ro) /certs/server from certs-server (ro) /run from host-run (rw) /sys/fs/cgroup from host-ctl (rw) /tmp from host-tmp (rw) /var/lib/buildkit from host-var-lib-buildkit (rw) /var/lib/rancher from host-containerd (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro) agent: Container ID: Image: rancher/kim:v0.1.0-beta.7 Image ID: Port: 1233/TCP Host Port: 1233/TCP Command: kim --debug agent Args: --agent-port=1233 --buildkit-socket=unix:///run/buildkit/buildkitd.sock --buildkit-port=1234 --containerd-socket=/run/k3s/containerd/containerd.sock --tlscacert=/certs/ca/tls.crt --tlscert=/certs/server/tls.crt --tlskey=/certs/server/tls.key State: Waiting Reason: CreateContainerError Ready: False Restart Count: 0 Environment: Mounts: /certs/ca from certs-ca (ro) /certs/server from certs-server (ro) /etc/pki from host-etc-pki (ro) /etc/ssl from host-etc-ssl (ro) /run from host-run (rw) /sys/fs/cgroup from host-ctl (rw) /var/lib/buildkit from host-var-lib-buildkit (rw) /var/lib/rancher from host-containerd (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: host-ctl: Type: HostPath (bare host directory volume) Path: /sys/fs/cgroup HostPathType: Directory host-etc-pki: Type: HostPath (bare host directory volume) Path: /etc/pki HostPathType: DirectoryOrCreate host-etc-ssl: Type: HostPath (bare host directory volume) Path: /etc/ssl HostPathType: DirectoryOrCreate host-run: Type: HostPath (bare host directory volume) Path: /run HostPathType: Directory host-tmp: Type: HostPath (bare host directory volume) Path: /tmp HostPathType: Directory host-var-lib-buildkit: Type: HostPath (bare host directory volume) Path: /var/lib/buildkit HostPathType: DirectoryOrCreate host-containerd: Type: HostPath (bare host directory volume) Path: /var/lib/rancher HostPathType: DirectoryOrCreate certs-ca: Type: Secret (a volume populated by a Secret) SecretName: kim-tls-ca Optional: false certs-server: Type: Secret (a volume populated by a Secret) SecretName: kim-tls-server Optional: false kube-api-access-f4cz4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/builder=true Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m36s default-scheduler Successfully assigned kube-image/builder-btlsg to k3d-kim-server-0 Normal Pulling 3m37s kubelet Pulling image "docker.io/moby/buildkit:v0.8.3" Normal Pulled 3m20s kubelet Successfully pulled image "docker.io/moby/buildkit:v0.8.3" in 16.8479912s Normal Created 3m19s kubelet Created container rshared-buildkit Normal Created 3m19s kubelet Created container rshared-tmp Normal Started 3m19s kubelet Started container rshared-tmp Normal Started 3m19s kubelet Started container rshared-buildkit Normal Pulled 3m19s kubelet Container image "docker.io/moby/buildkit:v0.8.3" already present on machine Normal Pulled 3m18s kubelet Container image "docker.io/moby/buildkit:v0.8.3" already present on machine Normal Created 3m18s kubelet Created container rshared-containerd Normal Started 3m18s kubelet Started container rshared-containerd Normal Pulling 3m17s kubelet Pulling image "rancher/kim:v0.1.0-beta.7" Warning Failed 3m17s kubelet Error: failed to generate container "73c7a50781eaf5de74ab1f95568a7bc7e26016fd4a06aa28bf8ea2a79be3f9dd" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount Normal Pulled 3m6s kubelet Successfully pulled image "rancher/kim:v0.1.0-beta.7" in 10.3655793s Warning Failed 3m6s kubelet Error: failed to generate container "c7913044e35bbb8ef948e2bd17848cb308888cb4dbeddc97d08bfad073d08853" spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/" but it is not a shared mount Warning Failed 3m6s kubelet Error: failed to generate container "4f9180c6190fedcd9601131d41b1ca48160330bbef6b40ca9b0fd2cbd0bae24c" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount Warning Failed 3m6s kubelet Error: failed to generate container "68a8bf3b5550081f11d07d1c5e614eb5024f65633d60b7a4d78913c15f10d091" spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/" but it is not a shared mount Warning Failed 2m55s kubelet Error: failed to generate container "356515b2435fc6ec28c8f3e7be405fc22a64d10b6b423ea17342e6cf30c7b823" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount Normal Pulled 2m55s (x2 over 3m6s) kubelet Container image "rancher/kim:v0.1.0-beta.7" already present on machine Warning Failed 2m55s kubelet Error: failed to generate container "0031efc0dc317b721e73d3b500b5b006730b16d0f354538cf5c7d728daafd802" spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/" but it is not a shared mount Normal Pulled 2m43s (x4 over 3m17s) kubelet Container image "docker.io/moby/buildkit:v0.8.3" already present on machine Warning Failed 2m43s kubelet Error: failed to generate container "4f60fc92c2b125fba4279390bfcd6a6255e4e0d8faec90838d49013b2c52b04a" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount ```

[tekumara]

$ kubectl -n kube-image logs builder-btlsg rshared-tmp
/tmp is a mountpoint
+ nsenter -m -p -t 1 -- env 'PATH=/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux' sh -c 'mount --make-rshared $_DIR'
mount: /tmp: Invalid argument

$ kubectl -n kube-image logs builder-btlsg rshared-buildkit
/var/lib/buildkit is a mountpoint
+ nsenter -m -p -t 1 -- env 'PATH=/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux' sh -c 'mount --make-rshared $_DIR'
mount: /var/lib/buildkit: Invalid argument

$ kubectl -n kube-image logs builder-btlsg rshared-containerd
/var/lib/rancher is a mountpoint
+ nsenter -m -p -t 1 -- env 'PATH=/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux' sh -c 'mount --make-rshared $_DIR'
mount: /var/lib/rancher: Invalid argument

[tekumara]

FYI I'm using k3s in k3d, and have the same issue on macos and ubuntu.

$ k3d --version           
k3d version v5.3.0
k3s version v1.22.6-k3s1 (default)

[ashlineldridge]

I am also seeing this issue. I'm trying to create the simplest k3d/k3s cluster and perform a kim build on it. If I switch to kind it works. I can reproduce using the following:

> k3d cluster create all-in-one -p 1233:1233@loadbalancer -p 1234:1234@loadbalancer
...

> ./bin/kim builder install --endpoint-addr 127.0.0.1
...
INFO[0074] Waiting on builder daemon availability...
INFO[0081] Waiting on builder daemon availability...
Error: timeout waiting for builder to become available

> kubectl get pods -n kube-image
NAME            READY   STATUS                 RESTARTS   AGE
builder-t6n7j   0/2     CreateContainerError   0          34m

If I look at the status of the failed pod, I see:

  containerStatuses:
  - image: rancher/kim:v0.1.0-beta.7
    imageID: ""
    lastState: {}
    name: agent
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'failed to generate container "90cbca9c4fc190572d3e0f8488aea95c26b0ef03310f2ac9f8431ca331359413"
          spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/"
          but it is not a shared mount'
        reason: CreateContainerError
  - image: docker.io/moby/buildkit:v0.8.3
    imageID: ""
    lastState: {}
    name: buildkit
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'failed to generate container "657848459e41fa6fa8f5d444deb509d370fa0f7163584e7a9353bf1380e213c3"
          spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not
          a shared mount'
        reason: CreateContainerError

My system information is as follows:

macOS: 12.3.1
kim: v0.1.0-beta.7 (e597b9564b47213734787b3e0c540a635b250bbf)
k3d: v5.4.3
k3s: v1.23.6-k3s1 (default)

[tekumara]

oh hai @ashlineldridge! 👋

[ashlineldridge]

@tekumara Hey Oliver! I didn't even recognise your username!

[dweomer]

I last tested this with k3d v4.4.3 without issue. While it is possible that something in the k3s image has changed I think it more likely that k3d 5.x has changed how /tmp is setup for the k3s container(s). It is possible that I have cheated in making this work on my dev box by modifying / to be rshared.