error building image: stream error: stream ID 13; NO_ERROR; received from peer

[Curtisjk]

Actual behavior As part of my dockerfile I am attempting to copy a directory from another image as follows:

COPY --from=internal-registry/timesten-client:22.1.1.22.0-java17 --chown=jetty:jetty /opt/timesten /opt/timesten

However, this results in the following error:

DEBU[0000] Found extra base image stage internal-registry/timesten-client:22.1.1.22.0-java17 
INFO[0000] Retrieving image manifest internal-registry/timesten-client:22.1.1.22.0-java17 
INFO[0000] Retrieving image internal-registry/timesten-client:22.1.1.22.0-java17 from registry internal-registry 
INFO[0002] Storing source image from stage internal-registry/timesten-client:22.1.1.22.0-java17 at path /kaniko/stages/internal-registry/timesten-client:22.1.1.22.0-java17 
error building image: stream error: stream ID 13; NO_ERROR; received from peer

This image does contain large layers (one is about 450MB) so I suspect that there is some issue with downloading the image, and it is silently failing.

When attempting to download the same image using docker itself, it results in a few retries to download some layers, but does eventually succeed after about 3 retries:

fe9c990952a7: Retrying in 4 seconds
d453be5a75b3: Download complete
4606aa215409: Retrying in 4 seconds
0090ad2bf7c6: Retrying in 5 seconds

Our registry is a privately hosted Harbor instance.

Expected behavior Better logging around why this occurred - is it related to failure to download the image?

Can Kaniko attempt to retry to download the layer like docker does?

I have verified that this works with smaller images to rule out any other registry issues (proxy/auth/etc.)

To Reproduce NA

Additional Information

• Dockerfile Please provide either the Dockerfile you're trying to build or one that can reproduce this error.

  FROM internal-registry/jetty:9.4.54-jre17-eclipse-temurin
  COPY --from=internal-registry/timesten-client:22.1.1.22.0-java17 --chown=jetty:jetty /opt/timesten /opt/timesten
  ENTRYPOINT ["/entrypoint.sh"]

• Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands) NA

• Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor:v1.22.0-debug

  Triage Notes for the Maintainers

  Description	Yes/No
  Please check if this a new feature you are proposing	- [ ]
  Please check if the build works in docker but not in kaniko	- [X]
  Please check if this error is seen when you use --cache flag	Not Attempted
  Please check if your dockerfile is a multistage dockerfile	- [ ]

[ntx-ben]

Happening to me as well from time to time using gcr.io/kaniko-project/executor:v1.23.2-debug.

Multi-stage Dockerfile which results in some large layers. Failing to upload to GitLab registry with Harbor as a mirror.