failed to push image with >32GiB layers

[gman0]

Description

Pushing an image containing layers with size exceeding 32GiB fails.

Logs:

• buildkitd:

  DEBU[2023-11-13T12:29:17+01:00] fetch response received                       digest="sha256:1aa43c883f27c10a667e7133f4fb38926a0657c24a039161837f9c8f3f1cc4e3" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.connection=keep-alive response.header.content-length=0 response.header.date="Mon, 13 Nov 2023 11:29:17 GMT" response.header.server=nginx response.header.set-cookie="sid=6bffe6455455725c19e41f437f45260f; Path=/; HttpOnly" response.header.x-request-id=5ee38247-3570-4480-9593-4468dcfd6a87 response.status="502 Bad Gateway" size=35444941962 spanID=cec4105e01fd1036 traceID=e14e4bbfe499709313bad2e1981ff279 url="https://rvasek-registry-debug-h3g6j6gocomo-master-0.cern.ch:30003/v2/library/big-1/blobs/uploads/c1e619a4-e43d-40cf-9bfa-812cb5625402?_state=ADeX_2mTiVkrHRTSNRaEpKIm4H0slxON-fpd1kEVUER7Ik5hbWUiOiJsaWJyYXJ5L2JpZy0xIiwiVVVJRCI6ImMxZTYxOWE0LWU0M2QtNDBjZi05YmZhLTgxMmNiNTYyNTQwMiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0xMS0xM1QxMToxMTowMy45MjkwNjU0OThaIn0%3D&digest=sha256%3A1aa43c883f27c10a667e7133f4fb38926a0657c24a039161837f9c8f3f1cc4e3"
  DEBU[2023-11-13T12:29:17+01:00] unexpected response                           body= digest="sha256:1aa43c883f27c10a667e7133f4fb38926a0657c24a039161837f9c8f3f1cc4e3" mediatype=application/vnd.oci.image.layer.v1.tar+gzip resp="&{502 Bad Gateway 502 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[0] Date:[Mon, 13 Nov 2023 11:29:17 GMT] Server:[nginx] Set-Cookie:[sid=6bffe6455455725c19e41f437f45260f; Path=/; HttpOnly] X-Request-Id:[5ee38247-3570-4480-9593-4468dcfd6a87]] {0xc000b8e060} 0 [] false false map[] 0xc00061e500 0xc00031eb00}" size=35444941962 spanID=cec4105e01fd1036 traceID=e14e4bbfe499709313bad2e1981ff279

• harbor-core Pod:

  2023/11/13 11:29:17 http: proxy error: readfrom tcp 10.100.207.18:45770->10.254.248.177:5000: 
  http: request body too large

• harbor-registry Pod:

  time="2023-11-13T11:29:18.09602447Z" level=error msg="client disconnected during blob PUT" auth.user.name="harbor_registry_user" contentLength=35444941962 copied=34359738368 error="unexpected EOF" go.version=go1.20.10 http.request.contenttype="application/octet-stream" http.request.host="rvasek-registry-debug-h3g6j6gocomo-master-0.cern.ch:30003" http.request.id=521901f6-59ee-4610-8c4c-3f3c1f2f6c32 http.request.method=PUT http.request.remoteaddr=10.100.132.0 http.request.uri="/v2/library/big-1/blobs/uploads/c1e619a4-e43d-40cf-9bfa-812cb5625402?_state=ADeX_2mTiVkrHRTSNRaEpKIm4H0slxON-fpd1kEVUER7Ik5hbWUiOiJsaWJyYXJ5L2JpZy0xIiwiVVVJRCI6ImMxZTYxOWE0LWU0M2QtNDBjZi05YmZhLTgxMmNiNTYyNTQwMiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0xMS0xM1QxMToxMTowMy45MjkwNjU0OThaIn0%3D&digest=sha256%3A1aa43c883f27c10a667e7133f4fb38926a0657c24a039161837f9c8f3f1cc4e3" http.request.useragent="buildkit/v0.0-dev" vars.name="library/big-1" vars.uuid=c1e619a4-e43d-40cf-9bfa-812cb5625402

(full logs here)

registry Pod prints an error with unexpected EOF at contentLength=35444941962 copied=34359738368; 34359738368 bytes is exactly 32GiB, i.e. 1 << 35. I've tracked this down to:

https://github.com/goharbor/harbor/blob/v2.9.1/src/core/main.go#L131

Changing this value to 1111111111 confirms that the upload fails due to this limit:

time="2023-11-13T16:56:29.009795711Z" level=error msg="client disconnected during blob PUT" auth.user.name="harbor_registry_user" contentLength=1258698756 copied=1111111111 error="unexpected EOF" go.version=go1.21.3 http.request.contenttype="application/octet-stream" http.request.host="rvasek-registry-debug-h3g6j6gocomo-master-0.cern.ch:30003" http.request.id=8ce841de-8067-4bda-a899-9d5b16e46a56 http.request.method=PUT http.request.remoteaddr=10.100.132.0 http.request.uri="/v2/library/big-2/blobs/uploads/3a28357f-7774-4c1d-b81b-b2277a0ef56c?_state=Sn3--urLlCGtulvxOJbH_pvgsb7QvmO3kUJDlZp2u457Ik5hbWUiOiJsaWJyYXJ5L2JpZy0yIiwiVVVJRCI6IjNhMjgzNTdmLTc3NzQtNGMxZC1iODFiLWIyMjc3YTBlZjU2YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0xMS0xM1QxNjo1NTo1NS4xNTgxNDgxMzlaIn0%3D&digest=sha256%3A9d0f378f076ea7045db9c04f5d574d85f5e3b6ed3be0f04f8a0fb564d7d31ef8" http.request.useragent="buildkit/v0.0-dev" vars.name="library/big-2" vars.uuid=3a28357f-7774-4c1d-b81b-b2277a0ef56c

Steps to reproduce the problem:

1. Building and pushing the image:
   1. Generate data-33gib file:

      dd if=/dev/urandom of=data-33gib bs=1M count=33792 status=progress

   2. Dockerfile:

      FROM alpine:latest
      COPY data-33gib /data-33gib
      RUN ls -lh /data-33gib

   3. Build and push with buildctl:

      buildctl --debug build                                                                                                                                                            \
      --frontend dockerfile.v0                                                                                                                                                         \
      --local context=.                                                                                                                                                                \
      --local dockerfile=.                                                                                                                                                             \
      --output type=image,name=rvasek-registry-debug-h3g6j6gocomo-master-0.cern.ch:30003/library/big-1:latest,oci-mediatypes=true,compression=estargz,force-compression=true,push=true \
      --import-cache type=local,src=/image-import-cache                                                                                                                                \
      --export-cache type=local,dest=/image-export-cache

2. Observe the push eventually fail (building the image with push=false succeeds)

Versions:

• harbor version: v2.9.1
• buildkitd: v0.12.3

Additional context:

Our user needs to create an image of around 100GiB total -- copying an application directory with many files as a single layer. It is possible to work around the 32GiB limit by splitting the layer into multiple ones by having more COPY instructions. However, because the directory contains hard-links, the total size of the image increases by 50%.

[chlins]

Duplicate with https://github.com/goharbor/harbor/issues/18888.