Kaniko hangs on multistage Dockerfile

javaDeveloperKid commented 8 months ago

Actual behavior Kaniko hangs on multistage build.

Expected behavior It does not hang.

To Reproduce This works:

FROM alpine AS base

RUN echo base

this does not:

FROM alpine AS base

RUN echo base

FROM base as prod

RUN echo prod

Additional Information

kaniko command:

$ /kaniko/executor --context . --dockerfile Dockerfile --destination repo1:tag1 --destination repo2:tag2 --digest-file my-digest-file` 
# ... 
INFO[0021] Pushing image to repo1:tag1
INFO[0022] Pushed repo1:tag1
INFO[0022] Pushing image to repo2:tag2
INFO[0022] Pushed repo2:tag2
# it hangs here

runs on Jenkins inside Kubernetes pod

See container configuration

  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
      - cat
    tty: true
    securityContext:
      privileged: true
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"
      limits:
        memory: "3Gi"
    env:
      - name: JENKINS_AGENT_WORKDIR
        value: /home/jenkins/agent
    volumeMounts:
      - mountPath: /var/lib/docker
        name: libcontainers
      - mountPath: /kaniko/.docker
        name: kaniko-secret

kaniko container running as root
```
$ id
uid=0 gid=0 groups=0
```
-v trace does not display anything after INFO[0022] Pushed repo2:tag2
docker inspect gcr.io/kaniko-project/executor:debug

See docker inspect details

[
    {
        "Id": "sha256:1598930707ba196c976f274b0844a5b1de3a73d9b0af386bd2bdf211747d5669",
        "RepoTags": [
            "gcr.io/kaniko-project/executor:debug"
        ],
        "RepoDigests": [
            "gcr.io/kaniko-project/executor@sha256:899886a2db1c127ff1565d5c7b1e574af1810bbdad048e9850e4f40b5848d79c"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2023-12-19T17:59:43.530057999Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/bin:/kaniko:/busybox",
                "HOME=/root",
                "USER=root",
                "SSL_CERT_DIR=/kaniko/ssl/certs",
                "DOCKER_CONFIG=/kaniko/.docker/",
                "DOCKER_CREDENTIAL_GCR_CONFIG=/kaniko/.config/gcloud/docker_credential_gcr_config.json"
            ],
            "Cmd": null,
            "Image": "",
            "Volumes": {
                "/busybox": {}
            },
            "WorkingDir": "/workspace",
            "Entrypoint": [
                "/kaniko/executor"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 96270058,
        "VirtualSize": 96270058,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/46b780dac9624473a0f8c31ba539f9fa20c6471fe764a6bc3c27bc514da4cb95/diff:/var/lib/docker/overlay2/bceee7154954aed071e599338f55db88964e12f27ad5013384773abc7e262f1f/diff:/var/lib/docker/overlay2/30ad7e7792533d741db9990d2efc154e649fe930125528e0b4e3e85c51c940bc/diff:/var/lib/docker/overlay2/56bb80ca96586b3197ec62e6236e9a48e399876b2dd0b937f91824d5acc7e9c0/diff:/var/lib/docker/overlay2/504e94cafcc90b5e5327028bb2a9aa76a5ad0528f3000f54a0b44ce0c9e32753/diff:/var/lib/docker/overlay2/50a533e417bc7b005280d973854cab3cf27c549b4f8ffeddbfe0393db86d858a/diff:/var/lib/docker/overlay2/d1bb66a82b05cc8803d1754ea55204c296a69a02fb2eafc4ecabd368c65fe673/diff:/var/lib/docker/overlay2/0326c1f3c63ab74449287400db3a1d0fbe94430cc02ef40381bc3bae17df85b9/diff:/var/lib/docker/overlay2/ee65b5674b50451865b77e1d03ed71b91b4587c0d6a0c6a10b4f7d97ca471133/diff:/var/lib/docker/overlay2/77a1fcd83a0c9f78002110af1249850828ecd7fd5745b96c849e3ff8579b0487/diff:/var/lib/docker/overlay2/99f428250ce1eb46826a31085a10acf660fad070acce389e24db4e9aa0ef8ac4/diff:/var/lib/docker/overlay2/05aa3d4b8cade3d468e25fef3088ffd946b5773f06824c72782a20f97a862b8c/diff",
                "MergedDir": "/var/lib/docker/overlay2/a58d6c9100349b1b68aaf199df38e6d634fa5314edf2737ea9e36d65dcc64649/merged",
                "UpperDir": "/var/lib/docker/overlay2/a58d6c9100349b1b68aaf199df38e6d634fa5314edf2737ea9e36d65dcc64649/diff",
                "WorkDir": "/var/lib/docker/overlay2/a58d6c9100349b1b68aaf199df38e6d634fa5314edf2737ea9e36d65dcc64649/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:acbb1d2131afbae2d9c93d14d2a32921f76e5cfc9557c489f8791d332bc39e6f",
                "sha256:8006edb0c67c2d3982a8a3e1345006a4d4f0389fa6033d9ecc1ee4aaad946b99",
                "sha256:7d2968491ac3ea5965534a6dbaa525e1708dd74cf8dfa58444f4bd9f7bbb44ae",
                "sha256:858516508d4ef2b518e150bddd3cf32ab8a62a179740c7047fb0e11919402158",
                "sha256:e28b7b203ccbd25247cc2a47ff2b6a50ea058d4ce4abf0c241ff7a1832aaa69f",
                "sha256:176a38e153439f5b6aa4866576313c046bb375dfe15803f09e627dc64826f744",
                "sha256:2685a07437e809f154112d52f40ef84e72b6673e06c93251abd4ffb8a02b59ee",
                "sha256:8cb586c688be2ca676a47bd449bcc797d5e60f9a9867f058486eac46555a8207",
                "sha256:19e97b7f1582933371357dab923af552615cd1ffd15bc875d28d1ac9bdba76b9",
                "sha256:804e5954d05669d51a6e1a26a825d03b80362aa085120e2e8570fabd3fe06bf3",
                "sha256:673fc86c31a79fa24babf910a7c29bcf3eaed0ee2e343aa8ca3b80fc002344a3",
                "sha256:39659de5db94cf47c109fd3f561ca84ef44c30c0b6782aa5bc2545fee71273c5",
                "sha256:aba7600bd40d84905ec2e55c60414d18ada1bdf6fdc0d5495af90b25dc6d89ef"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

The following works (destination is a repo without authentication):

$ docker run --rm -it --entrypoint='' gcr.io/kaniko-project/executor:debug /bin/sh
$ pwd # outputs /workspace
$ echo -e "FROM alpine AS base\nRUN echo base\nFROM base as prod\nRUN echo prod" > Dockerfile 
$ /kaniko/executor --destination <repo-without-authentication>/foo

I think I read somewhere that kaniko may be removing the whole filesystem between stages so the mounted docker registry auth credentials will be removed after the first stage.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use `--cache` flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [x]

martincomputershare commented 5 months ago

Same symptoms here. Using Kaniko v1.22.0-debug and Jenkins.

babs commented 1 month ago

Have you tried to increase kaniko's verbosity to trace to see what action makes it hang ?

Mosaad-20 commented 3 weeks ago

Same issue here, I have tried increasing the verbosity level to trace and there is no log after pushing the image

Update: if you are using cri-o try this fix: #1275

GoogleContainerTools / kaniko

Kaniko hangs on multistage Dockerfile #2985