Spike: Can we take a rootfs.tgz and run it in K8s without uploading it to a registry

[chenbh]

One of the ways Concourse creates containers is directly from a rootfs.tgz. If we want to consider https://github.com/concourse/rfcs/pull/22 we would need a way to get k8s to create and run our image without having to upload it to a registry

• [x] Given an exported image (OCI compliant or via docker save) that is available to the pod, run it in k8s without using a registry
• [x] Given a rootfs (e.g. via docker export) that is available to the pod, run it in k8s without using a registry
• [x] Figure out how image caching in k8s work, how long do they last? Do we have to clear cache manually?
• [ ] Explore non-docker container runtimes

[chenbh]

Turns out it wasn't hard at all. It just involves init containers and mounting the docker socket of the host (node) to the container. We can then use the docker CLI from inside the container to communicate with the daemon running on the node.

Notes:

• the image.tgz or rootfs.tgz must be available to the init container already (related: https://github.com/concourse/concourse/issues/3740)
• the custom image cannot be used by other init containers (they're all started in parallel so it just can't work)
• imagePullPolicy: IfNotPresent is important as otherwise it might try to pull first

apiVersion: batch/v1
kind: Job
metadata:
  name: offline-images
spec:
  template:
    spec:
      initContainers:
      - name: upload-it
        image: gcr.io/cf-concourse-production/uploader
        command: [ "docker", "import", "/rootfs.tgz", "local-image" ]
        # command: [ "docker", "image", "load", "-i", "/image.tgz" ]
        volumeMounts:
        - name: dockersock
          mountPath: /var/run/docker.sock

      containers:
      - name: use-it
        image: local-image
        imagePullPolicy: IfNotPresent
        command: [ "sh", "-c", "ls -l /boo" ]

      restartPolicy: Never
      volumes:
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock

Further exploration

What about other k8s container engines (https://kubedex.com/kubernetes-container-runtimes/)?

[chenbh]

Figure out how image caching in k8s work, how long do they last? Do we have to clear cache manually?

Should be built in: https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/