search

tilt-dev / tilt

Define your dev environment as code. For microservice apps on Kubernetes.
https://tilt.dev/
Apache License 2.0
7.71k stars 305 forks source link

team members can share image build artifacts #3629

Open nicks opened 4 years ago

nicks commented 4 years ago

In the #tilt channel, Adam writes:

Hi folks; I’m trying to speed up tilt {up|ci} runs on fresh checkouts by re-using previously build images. To wit:
A fresh git checkout of a branch on a new machine without any previous runs of tilt,  should, on a tilt up or tilt ci, try to pull images from a remote repo and only rebuild those images if needed.

There are a couple different possible recommendations on this. Some teams we work with have fairly complex Tiltfiles that try to simulate version of this (e.g., "pass this environment variable to get the pre-built version of service X, Y, and Z")

Docker build has a --cache-from parameter https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources

but it takes some wiring to get this to work well with tilt (both to use the cache and to populate the cache)

This is a pretty common thing to want to do, but we should research how to do it and provide recommendations

nicks commented 4 years ago

Currently, docker_build has a cache_from parameter, which should make it easier to play around with solutions in a Tiltfile (though ultimately I'd want something that auto-configured a bit better without you having to know it exists)

nicks commented 4 years ago

(you'd also have to set up tilt to set build_args={'BUILDKIT_INLINE_CACHE': '1'} and push the image to a remote repo)

nicks commented 4 years ago

Akshat Khandelwal in the Tilt channel pointed out this approach, which i thought was neat:

We have a bunch of docker_build calls like below -

docker_build(
   'my-image',
   '.',
   dockerfile='./service/Dockerfile',
   build_args = {'BUILDKIT_INLINE_CACHE': '1'},
   cache_from = ['my-image:<HEAD commit id>', 'my-image:<HEAD~1>', 'my-image:master commit id', 'my-image:master~1'],
   extra_tag = 'my-image:<HEAD commit id>'
)

Where currently, the values of HEAD commit id, master commit id etc. are calculated at the start of every Tilt run. The idea is to use external cache sources, and use multiple 'commit id' tags to increase chances of getting cache hits.

i'm not 100% confident on the semantics of cache_from and how it pulls the cache from the remote registry, though i wonder if tilt could do something intelligent here about selecting caches

adamf commented 4 years ago

@nicks Akshat is working with us :)

He/we will have some updates on our caching progress; we've basically got it working well but have had to do some tricks with making the tilt build context clean from local files that aren't meant to be in the docker images but can't easily be docker/git ignored. FWIW, this saves us an insane amount of build time for all our developers and in CI.