search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.21k stars 1.16k forks source link

Support pulling individual files without pulling entire image #1502

Open tomwillfixit opened 4 years ago

tomwillfixit commented 4 years ago

Creating this ticket for visibility and based on a conversation with @thaJeztah

image

Use Case

The use case for this enhancement is that I'm using multi stage builds and copying internal tools (usually binaries) from various images owned by other teams. Currently I just use the "COPY --from" directive and on a fresh GitLab Runner with no build cache this will lead to the whole image being pulled for each tool (binary) I'm trying to copy. This can lead to multiple GBs being pulled.

I'm currently using this CLI plugin as a workaround : https://github.com/tomwillfixit/docker-artifact

Exposing individual files as artifacts through labelling or some sort of internal manifest would allow more granular operations on files.

Feel free to close this out as I'm aware that there is a "stargz" ticket which may offer cover this use case. Just wanted to put this on the radar.

Thanks.

thaJeztah commented 4 years ago

Thanks!

/cc @AkihiroSuda @ktock

ktock commented 4 years ago

@tomwillfixit Thanks! There is an on-going PR related to "lazy pull" on https://github.com/moby/buildkit/pull/1475. Maybe we need some follow-up patches for fully supporting this use-case and I'm willing to work on it after this PR get merged!

tomwillfixit commented 4 years ago

Thanks very much for the quick response! Happy to help out in any way I can with testing etc in future.