Open jcushman opened 2 years ago
OK, I have what I think is bad news. Although the issues aren't perfectly clear, I believe that the Harbor registry does not recognize the format of the build cache.
https://github.com/goharbor/harbor/issues/14774 https://github.com/goharbor/harbor/issues/16624
This may be true of other registries as well:
As a side note, we use DockerHub as the cache repository instead of Google Container Registry which stores our application images. This is because at the time of writing, Google Container Registry does not seem to support the cache manifest format application/vnd.buildkit.cacheconfig.v0 and returns Bad Request 400 when trying to push a build cache. So we fell back on using a private repo on DockerHub for now and it works perfectly.
The evidence here is from running
docker buildx bake -f docker-compose.yml -f docker-compose.override.yml web --push
with a minimal docker-compose file, where docker-compose.override.yml
includes
x-bake:
tags:
- registry.lil.tools/harvardlil/caching-test:0.01
platforms:
- linux/amd64
- linux/arm64
cache-from:
- type=registry,ref=registry.lil.tools/harvardlil/caching-test:buildcache
cache-to:
- type=registry,ref=registry.lil.tools/harvardlil/caching-test:buildcache,mode=max
Unsurprisingly, we see
=> ERROR importing cache manifest from registry.lil.tools/harvardlil/caching-test:buildcache 0.4s
but then at the end we see
=> => writing manifest sha256:d7f3744244341356ea9448a0a46006cd119197af12eac69ad378b6ab54a22317 0.2s
------
> importing cache manifest from registry.lil.tools/harvardlil/caching-test:buildcache:
------
------
> exporting cache:
------
error: failed to solve: error writing manifest blob: failed commit on ref "sha256:d7f3744244341356ea9448a0a46006cd119197af12eac69ad378b6ab54a22317": unexpected status: 404 Not Found
which matches https://github.com/goharbor/harbor/issues/14774.
https://github.com/opencontainers/image-spec/issues/870 has
... There are a few intereseting examples . One is buildkit cache which actually usese descriptors which are blobs and not a oci image manifest. In ACR we went ahead and supported this use case for buildkit.
I believe #8 finally obviates the need for this kind of caching.
Under correction from @rebeccacremona I am reopening this; #8 solves our problem of building and pushing an image too frequently, but does not accomplish the caching of the layers that make up the image. For future reference, a Harbor update might allow us to use a cache manifest, or we might switch to another registry.
https://github.com/harvard-lil/docker-compose-update-action/commit/750f46aa actually solves the problem of building and pushing too frequently; docker manifest inspect
was not working in CI.
These cache flags are wrong:
The way it wants to work is to cache-from the registry (in all cases), and cache-to the registry (if logged into the registry).