GoogleContainerTools / kaniko

Build Container Images In Kubernetes
Apache License 2.0
14.64k stars 1.42k forks source link

Kaniko - 2nd destination directory is not created when using Cloud Build #2817

Open dazdaz opened 11 months ago

dazdaz commented 11 months ago

The 2nd destination folder does not create a cache folder on Google Cloud Artifact Registry. I am using the latest kaniko release.

Only the first cache directory is created from my cloudbuild.yaml.

By swapping the destinations around, I can confirm that the 2nd destination folder is not created on Artifact Registry.

I've read around the issues on GitHub and the general consensus appears to be that a 2nd destination cache is possible but it just doesn't work for me in my tests.

Is a 3rd, 4th and 5th cache directory supported with Kaniko or limited to 2 cache folders ?

Here is the relevant section in cloudbuild.yaml

Both container images are created successfully and pushed to Artifact Registry however the 2nd destination directory is not created

This is the directory structure that I see on Artifact Registry, so app2-image-cache has not been created.

app1-image
app1-image/cache
app2-image
steps:
    - name: 'gcr.io/kaniko-project/executor:latest'
      args: [
      "--destination=${_MYREGION}-docker.pkg.dev/${_PROJECT_ID}/quickstart-docker-repo/app1-image:${BUILD_ID}",
      "--destination=${_MYREGION}-docker.pkg.dev/${_PROJECT_ID}/quickstart-docker-repo/app2-image:${BUILD_ID}",
      "--cache=true",
      "--cache-ttl=8h"
      ]
JeromeJu commented 10 months ago

Thanks for filing this @dazdaz , this does look like a bug within the implementations of cache being pushed to the 2nd destination as specified. (Thanks the pointers from @aaron-prindle 🙏 ) The normal use case seems to be with one destination specified. In order to find the best way solving this issue, can I ask for what's the use case for doing so for using multiple destinations for caching?

dazdaz commented 10 months ago

Large microservice based app, with a lot of different images requiring some form of organization through the use of more than 1 repo due to different owners designated to each repo. I'd presume that some folks would require more than 3, 4, 5 etc

Isn't this a simple case of mkdir() missing for the secondary destination or is there more to it ?

Or do you envisage a design whereby there may well be 2,3,4,5 repo's but only 1 shared container cache, if so then this could have security implications where companies would not use it, whereas individualised cache for each repo would be ok.