Storing source image from stage 0: unexpected EOF

[chris-morandi]

Actual behavior Kaniko build failing, approximately 30% of the time at stage 0 when storing the image, hanging for up to 10 minutes then reporting an "unexpected EOF"

Expected behavior All stages build and the image is pushed to the destination registry

To Reproduce This is an intermittent error. Which occured when switching registries.

Additional Information We are building in a kubernetes cluster 1.21 using docker version 20

We had not see this error until moving to a new mirror and destination registry. We are a bit confused with the error as the log line indicates INFO[0050] Storing source image from stage 0 at path /kaniko/stages/0. So unless this is a timing issue we can't see why how storing and the registry are related.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[rafagsiqueira]

We are seeing the same issue. Using kaniko executor:v1.8.1-slim.

[nejch]

Has anyone had any luck getting to the bottom of this? :)

[bat553]

Same issue here, but the job continue in the background.

Environment :

• Kubernetes runner : drone/drone-runner-kube:1.0.0-rc.3
• Kaniko : latest
• Drone 2.16.0

[aaron-prindle]

Thanks for flagging this @chris-morandi. Can anyone in the thread here provide a series of steps to reproduce this issue, perhaps repro project + steps or some additional context on how they are able to trigger this? Thanks!

[SuperLight-007]

@aaron-prindle @chris-morandi we get the same issiue. 1.case by node disk or harbor io 2.case by memory fragmentation，when we build if the base image lager than 30G. log like this in os Jun 8 11:04:35 node-1 kernel: runc:[1:CHILD]: page allocation failure: order:6, mode:0xc0d0 Jun 8 11:04:35 node-1 kernel: kmem_cache_create(nf_conntrack_48971) failed with error -12 ...... Jun 8 11:04:35 node-1 kernel: Unable to create nf_conn slab cache

[aaron-prindle]

Not sure if related but an issue w/ "unexpected EOF" was reported here: https://github.com/GoogleContainerTools/kaniko/issues/2589

That issue is related to using the warmer though which it isn't clear to me from the above if that was used here as well

[andre161292]

We get a similar error from time to time, but not at stage 0. The first time we faced the issue was something like two weeks ago, being on the kaniko-project/executor:debug tag.

We're running two builds in parallel on the same sources. I'll make those sequential and will give you an update if that works out better.

We're using Kaniko as a container in a Kubernetes Pod. Here's the output of the Jenkins pipeline:

[2023-09-07T09:00:06.063Z] + /kaniko/executor '--dockerfile=Dockerfile' '--context=dir:///home/jenkins/agent/workspace/APP_app-rust_PR-70' --log-timestamp '--registry-mirror=docker-hub.intranet.io' --reproducible '--build-arg=BINARY_NAME=myapp' --no-push
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Using dockerignore file: /home/jenkins/agent/workspace/APP_app-rust_PR-70/.dockerignore 
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Retrieving image manifest docker-registry.intranet.io/base-images/debian:11-slim 
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Retrieving image docker-registry.intranet.io/base-images/debian:11-slim from registry docker-registry.intranet.io 
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Built cross stage deps: map[]                
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Retrieving image manifest docker-registry.intranet.io/base-images/debian:11-slim 
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Returning cached image manifest              
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Executing 0 build triggers                   
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Building stage 'docker-registry.intranet.io/base-images/debian:11-slim' [idx: '0', base-idx: '-1'] 
[2023-09-07T09:00:06.063Z] INFO [2023-09-07T09:00:05Z] Unpacking rootfs as cmd COPY target/x86_64-unknown-linux-gnu/release/$BINARY_NAME /app requires it. 
[2023-09-07T09:00:08.625Z] INFO [2023-09-07T09:00:08Z] ARG BINARY_NAME                              
[2023-09-07T09:00:08.625Z] INFO [2023-09-07T09:00:08Z] COPY target/x86_64-unknown-linux-gnu/release/$BINARY_NAME /app 
[2023-09-07T09:00:08.883Z] INFO [2023-09-07T09:00:08Z] Taking snapshot of files...                  
[2023-09-07T09:00:13.066Z] INFO [2023-09-07T09:00:12Z] ENV RUST_LOG=info                            
[2023-09-07T09:00:13.066Z] INFO [2023-09-07T09:00:12Z] CMD ["/app"]                                 
[2023-09-07T09:00:21.248Z] error building image: setting layer times: writing layer file: unexpected EOF
script returned exit code 1