MarkEdmondson1234 / googleCloudRunner

Easy R scripts on Google Cloud Platform via Cloud Run, Cloud Build and Cloud Scheduler
https://code.markedmondson.me/googleCloudRunner/
Other
81 stars 26 forks source link

Is it possible to add an argument for kaniko_cache = TRUE to cr_deploy_plumber? #132

Closed dcaud closed 2 years ago

dcaud commented 2 years ago

I've got a build that is taking a long time. It seems one way to speed up re-builds is to cache. Any ideas on how to make that work?

MarkEdmondson1234 commented 2 years ago

The kaniko cache applies to Docker containers, if that's a long build you can add the flag to the Docker build step so the second build speeds up.

Order of commands in your Docker container also matters, try to put commands that change more often later in the build eg the R script copy into the build.

You can also try building an intermediate container in another build that you FROM for your plumber script.

dcaud commented 2 years ago

Thanks for the tips. It seems some of the caching is already happening (via whatever Cloud Build does), but I will investigate some of your tips.

MarkEdmondson1234 commented 2 years ago

If all you are changing is the R script it should only take a couple of minutes.

dcaud commented 2 years ago

I'm not quite sure how the caching works -- but on the first re-build in a couple months, the whole docker image had to be rebuilt. However, subsequent re-builds used Google's caching. Maybe the first re-build I did had something odd with it. Either way, things work well at the moment.

MarkEdmondson1234 commented 2 years ago

The cache default timeout is 2 weeks. You can configure that with cache-ttl if working with directly https://cloud.google.com/build/docs/kaniko-cache#configuring_the_cache_expiration_time

2 weeks sounds about right though, as the dependencies have probably changed in that time period. If you want to fix the Docker image in that state, tag it and use that tag for your Docker pulls, not :latest (default when no tag)

MarkEdmondson1234 commented 2 years ago

And yes, I've looked to add kaniko_cache=TRUE to all the deployment functions that depend on the cr_buildstep_docker() function.