Open waveywaves opened 2 years ago
How big is the layer you're trying to copy? Do you have this image on another registry like docker hub so we can try to reproduce?
Image information and size of each layer
> Run docker image ls ttl.sh/0f030c5a-7565-4821-af90-5e83ca59b0cf:2h
REPOSITORY TAG IMAGE ID CREATED SIZE
ttl.sh/0f030c5a-7565-4821-af90-5e83ca59b0cf 2h 656890bddeb1 45 seconds ago 999MB
IMAGE CREATED CREATED BY SIZE COMMENT
656890bddeb1 45 seconds ago CMD ["/docker-entrypoint.sh"] 0B buildkit.dockerfile.v0
<missing> 45 seconds ago COPY app-config.yaml ./ # buildkit 3.77kB buildkit.dockerfile.v0
<missing> 45 seconds ago RUN /bin/sh -c tar xzf bundle.tar.gz && rm b… 35.4MB buildkit.dockerfile.v0
<missing> 45 seconds ago COPY /app/packages/backend/dist/bundle.tar.g… 8.74MB buildkit.dockerfile.v0
<missing> 46 seconds ago RUN /bin/sh -c tar xzf skeleton.tar.gz && rm… 484MB buildkit.dockerfile.v0
<missing> About a minute ago COPY /app/yarn.lock /app/package.json /app/p… 903kB buildkit.dockerfile.v0
<missing> 3 minutes ago ENV NODE_ENV=production 0B buildkit.dockerfile.v0
<missing> 3 minutes ago WORKDIR /app 0B buildkit.dockerfile.v0
<missing> 3 minutes ago USER node 0B buildkit.dockerfile.v0
<missing> 3 minutes ago RUN /bin/sh -c chmod +x docker-entrypoint.sh… 280MB buildkit.dockerfile.v0
<missing> 4 minutes ago COPY packages/backend/docker-entrypoint.sh .… 245B buildkit.dockerfile.v0
<missing> 13 days ago /bin/sh -c #(nop) CMD ["node"] 0B
<missing> 13 days ago /bin/sh -c #(nop) ENTRYPOINT ["docker-entry… 0B
<missing> 13 days ago /bin/sh -c #(nop) COPY file:4d192565a7220e13… 388B
<missing> 13 days ago /bin/sh -c set -ex && savedAptMark="$(apt-… 9.5MB
<missing> 13 days ago /bin/sh -c #(nop) ENV YARN_VERSION=1.22.19 0B
<missing> 13 days ago /bin/sh -c ARCH= && dpkgArch="$(dpkg --print… 100MB
<missing> 13 days ago /bin/sh -c #(nop) ENV NODE_VERSION=16.17.0 0B
<missing> 13 days ago /bin/sh -c groupadd --gid 1000 node && use… 333kB
<missing> 13 days ago /bin/sh -c #(nop) CMD ["bash"] 0B
<missing> 13 days ago /bin/sh -c #(nop) ADD file:5bd53bff884e470b3… 80.5MB
the largest layer is 484MB.
You can check the github action where I am running the above command over here. I also attempt to push the image to ttl.sh after, after which it fails.
I found a related issue where the user is facing the same problem. https://github.com/docker/build-push-action/issues/634#issuecomment-1157026269 I see that there is a cloudflare folder present in the ttl.sh repo as well. So I believe this requests are gtting proxied through cloudflare. The following excerpt from the above comment might be of some help to fix this.
The harbor domain was proxied through Cloudflare, disabling it finally let the
action push the image successfully. I guess it make sense for CF to return
413 because I never got it in my nginx logs. It feels weird though, why did
building and pushing through the cli succeed? Did I just stumble into an
edge case where manual build produced a layer within CF's limits while
the github builder went over?
Can you try this from another location? I suspect your error is coming from some kind of proxy between you and the ttl.sh
service. When I try to push a 1gb layer it seems to work fine.
─ docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 6821543d4f57 53 seconds ago 1.05GB
╰─ docker tag 6821543d4f57 ttl.sh/iz-test:1hr
╰─ docker push ttl.sh/iz-test:1hr
The push refers to repository [ttl.sh/iz-test]
23d2a9cb459a: Pushed
7fb796040f6b: Pushed
1hr: digest: sha256:ace2775a9f8d29121c9ba814e8b3a36bf7567fcfbb999baa335619cbdc07befe size: 740
╰─ docker history ttl.sh/iz-test:1hr
IMAGE CREATED CREATED BY SIZE COMMENT
6821543d4f57 7 minutes ago COPY ./bigfile /var/bigfile # buildkit 1.05GB buildkit.dockerfile.v0
<missing> 12 days ago /bin/sh -c #(nop) CMD ["sh"] 0B
<missing> 12 days ago /bin/sh -c #(nop) ADD file:e0416d710749b8a1d… 1.38MB
Here is my Dockerfile if you want to try to reproduce:
from busybox
copy ./bigfile /var/bigfile
I used dd if=/dev/urandom of=./bigfile bs=1M count=1000
to make the 1gb file.
I was able to push a bigger image as well. Did you check ? https://github.com/replicatedhq/ttl.sh/issues/104#issuecomment-1257873143
Can you share the url to the heroku instance where you have ttl.sh deployed ? I would like to go ahead and push there directly to bypass cloudflare.
I am hitting this issue in a repeatable GitHub actions. Feel free to try there.
Hi @kaovilai I spoke with @waveywaves on Kubernetes slack. I believe this because the buildx plugin is using the the push driver from containerd which apparently does not support transfer-encoding: chunked
see:
https://github.com/containerd/containerd/blob/main/remotes/docker/pusher.go#L265
The best workaround for now is just to use regular docker push
instead of using buildx to push your image. I do see some other code in the buildx code for using a moby
driver which might as well, but I'm not sure how to enable it through the github action.
I did some testing, and it's definitely the containerd push driver that is not chunking. I can reproduce by doing
./ctr i pull docker.io/library/openjdk:20-jdk-slim-buster --all-platforms
./ctr image tag docker.io/library/openjdk:20-jdk-slim-buster ttl.sh/iz-test-123:1hr
./ctr --debug image push ttl.sh/iz-test-123:1hr
Result:
DEBU[0001] unexpected response body="<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" digest="sha256:8c3a8eecab48f4c2c2053b014c1ce723acdbee8fb8d8831002f1672bb3db6843" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip resp="&{413 Request Entity Too Large 413 HTTP/1.1 1 1 map[Alt-Svc:[h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400] Cf-Ray:[752dbafd2a9c872f-ORD] Connection:[keep-alive] Content-Type:[text/html] Date:[Fri, 30 Sep 2022 14:43:21 GMT] Nel:[{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}] Report-To:[{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=iipZHB6T8KSIk4ebIOT5oRyCzUx2hFu1xZhuURFrx0rx9nW6hkSkLNvQuKslhpwHSqI27EHj27O53lNSpvhdFHWZPIXsCIBrdbPUkHF6eSumf4rUpuRX5cg%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}] Server:[cloudflare]] 0x14000702040 -1 [chunked] false false map[] 0x14000208000 0x1400053a0b0}" size=197212069
I submitted a PR to fix this in the upstream buildx uses: https://github.com/containerd/containerd/pull/7459
I am getting an error while pushing to ttl.sh,
413 request entity too large
. https://github.com/waveywaves/backstage/actions/runs/3103852916/jobs/5027642820#step:16:661Based on the docker documentation I understand that the following configuration is necessary to get around this issue https://docs.docker.com/registry/recipes/nginx/#setting-things-up