Runaway container size when building an image

[cmamigonian]

I'm building a fairly large Docker image (~9GB) in a k8s pod:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: img-claim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 80Gi
  storageClassName: gp2-encrypted-delete

---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: img
  name: img
  annotations:
    container.apparmor.security.beta.kubernetes.io/img: unconfined
    container.seccomp.security.alpha.kubernetes.io/img: unconfined
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  initContainers:
    - name: git-clone
      image: alpine/git:v2.26.2
      args:
        - clone
        - --single-branch
        - https://domain.com/path/to/repo
        - /repo # Put it in the volume
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      volumeMounts:
        - name: shared-volume
          mountPath: /repo
          subPath: repo
  containers:
  - image: r.j3ss.co/img:v0.5.11
    imagePullPolicy: Always
    name: img
    env:
      - name: DOCKER_CONFIG
        value: /img/.docker
    resources:
      limits:
        cpu: 1
        memory: 1Gi
      requests:
        cpu: 200m
        memory: 512Mi
    workingDir: /repo
    command:
    - img
    - build
    - --target=prod
    - --cache-to=type=registry,ref=my-registry/repo:cache
    - --cache-from=my-registry/repo:cache
    - -t
    - my-registry/repo:tag
    - -o
    - type=image,name=my-registry/repo:tag,push=true
    - <my-context>
    volumeMounts:
    - name: shared-volume
      mountPath: /repo
      subPath: repo
    - name: img-secret
      mountPath: /img/.docker
    - name: shared-volume
      mountPath: /tmp
      subPath: tmp
  restartPolicy: Never
  volumes:
  - name: img-secret
    secret:
      secretName: regcred
      items:
        - key: .dockerconfigjson
          path: config.json
  - name: shared-volume
    persistentVolumeClaim:
      claimName: img-claim

The pod ends up failing because the node runs out of space, and it's due to the img container growing to be well over 40GB (my nodes are 50GB EBS volumes). I see even trivial COPY commands of single files take forever to complete and the container will grow several GB in that time.

Granted I'm a novice with k8s and completely new to img, but am I doing something dumb here, or is there something underlying with img that's an issue?

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

[cmamigonian]

As I've done more investigation, the overlay filesystem is what ends up filling up. I'm trying to see where, but when I do a du -sh in / I get permission denied on a bunch of the /proc/ subdirectories. Is that a clue?

[cmamigonian]

After more research it appears this is an issue underlying Buildkit in that the entire filesystem is snapshotted on each layer. Can anyone else confirm this is the case?

[dlundgren]

I'm seeing a similar issue when building in a Gitlab CI pipeline using docker+machine with alpine:latest base docker image.

[dlundgren]

After debugging I found that we were using the native back-end (which copies each layer), and had a large layer early in our build was causing our problem. Once we squashed the base image, the build worked.

In our case, our 3rd layer was 4.8GB, and we had 8 layers after this, and with only 25GB available on our build instance, we were running out of space when the 9th layer was being applied.

[cmamigonian]

Closing this as using Ubuntu nodes with overlayfs solved the problem.