Caching doesn't seem to work with v5

[viraptor]

I've just tried to upgrade the plugin from 4.16 to 5 in a build where the caching worked correctly with v4. The step has a config like:

build:
  web
cache-from:
  - web:12345.ecr....amazonaws.com/app:latest-test
push:
  - web:12345.ecr....amazonaws.com/app:latest-test

In the buildkite output, I've got:

#5 [internal] load build context
#5 DONE 0.0s
#6 [ 1/15] FROM docker.io/library/ruby:3.2.1-slim-buster@sha256:8a0d9c234b69593c19f66658bccde67845681bc40c4b75b5767b74b5dd8adaad
#6 DONE 0.0s
#7 importing cache manifest from 12345.ecr....amazonaws.com/app:latest-test
#7 ...
#8 [auth] sharing credentials for 12345.ecr....amazonaws.com
#8 DONE 0.0s
#7 importing cache manifest from 12345.ecr....amazonaws.com/app:latest-test
#7 DONE 2.4s

But the cache is never used. I can actually see the FROM image being downloaded in the next step, even though it should be included in the cached image already.

[toote]

The log lines that say importing cache manifest do indicate that the caching is working. Note that, as the documentation states, the images you are trying to use as cache were built and pushed with BUILDKIT_INLINE_CACHE=1. Can you confirm that is the case? Even if so, if they are not being used as cache it means that docker decided not to and may have nothing to do with the plugin itself.

Finally, you mention that the image is being downloaded in the next step (which I assume is the same buildkite step, just the next step of the docker build process). In previous versions of the plugin, the cache-from option actually did a docker pull of the images, whereas now they are just mapped to docker's --cache-from option which should only do the pull if and when docker decides it would be beneficial to. As far as I understand from docker's behaviour when the --cache-from option is used, the manifest being downloaded is just a small description of the layers contained in the image so the build engine can decide if it can re-use them. The actual downloading happens afterwards, which appears to be what is happening here.

We will be doing some tests to review that they are actually being used correctly but any further information you can provide on what you are seeing and what you expect to see will be greatly appreciated.

[viraptor]

Aha! No, they're are not setting the inline cache. It looks like the new cli that came with v5 defaults to buildkit, which changed the behaviour. That makes sense.