Precache images using snapshots

[runningman84]

Description

What problem are you trying to solve? For some ml usecases we are dealing with quite big docker images. It would be cool to somehow put the images in an extra volume and advice karpenter to attach a volume based on some snapshot containing all these big images. This would greatly reduce data transfer costs.

How important is this feature to you?

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[tzneal]

It would take some work, but you should be able to do this now.

1) Create a provisioner that launches a node with an extra volume using block device mappings 2) Use custom user data to mount that volume to where container images are stored 3) Pull any desired images to the node 4) Create a snapshot of the volume 5) Update the block device mapping to specify the snapshotID

[FernandoMiguel]

I have a similar discussion topic for bottlerocket https://github.com/bottlerocket-os/bottlerocket/discussions/3477

my concern with this approach is that there will be container data, not only the container image in the snapshot

[tzneal]

my concern with this approach is that there will be container data, not only the container image in the snapshot

You could do some manual cleanup between 3 & 4, but would need to detach the instance from Karpenter and drain it. There's a draft upstream KEP for splitting the readonly/readwrite image filesystem that would make this easier at https://github.com/kubernetes/enhancements/pull/4198

[FernandoMiguel]

thanks for that link @tzneal . i'll subscribe to it.