Better size estimation

[zvin]

Fixes #231

[dfunckt]

I'm not sure about this change @zvin, it isn't fully addressing the linked issue (results are pretty much arbitrary) and at the same time gives worse results for the most common use case.

This is because the API stores the size of the image on disk (so the old code was accurate for the common use case), whereas the registry will reply with the amount of bytes that will be downloaded. Apparently you need the former, and the only way to get an accurate result without counting shared layers multiple times is to actually pull the images and query the layer sizes on disk via the local daemon.

[zvin]

@dfunckt why do you think the results are arbitrary ? We sum the sizes of tared layers. It will be slightly larger (around 3% in my tests) than the size of the layers on the disk.

Today, the common use case is multi-container apps. And most multi-container apps share some layers. Here https://jel.ly.fish/847be677-7404-4c60-8fa6-b6344b4f4ae8 the user complains that we create a 19GiB image for a device with 16GiB storage.

[pdcastro]

the user complains that we create a 19GiB image for a device with 16GiB storage.

Shaun summarised it in this CLI issue: https://github.com/balena-io/balena-cli/issues/2043

[dfunckt]

Results are just an estimate @zvin based on your heuristic that tar'd layers end up 3% larger. Can't we retrieve absolute numbers? What's wrong with pulling the images and doing the calculation locally? You are going to pull the images later on, right?

Also, I think the problem where we may end up with an image that is larger than what would fit on the device is orthogonal and still has to be handled regardless of how this PR ends up.

[zvin]

What's wrong with pulling the images and doing the calculation locally?

We need a place to pull them into.

[dfunckt]

I saw you mentioned resize2fs in FD -- would creating space large enough to hold the images as before and then calculating the actual size and resizing it down work? Otherwise, can you not create temporary space and copy the layers over when done?

[dfunckt]

@pdcastro described it really nicely in that issue thread:

Perhaps a solution might be to add a "shrinking" step after successful preloading, reducing the amount of free disk space in the image file. This would require "disk defragmenting", but I think there are some Linux tool that can achieve it.

[zvin]

would creating space large enough to hold the images as before and then calculating the actual size and resizing it down work?

It would, this PR still stands though. We need to estimate the required space for pulling the images. Imagine an application with 100 slightly different containers sharing a 2GiB layer. With this PR we will allocate ~2GiB, before this PR we would allocate 200GiB.

Otherwise, can you not create temporary space and copy the layers over when done?

I guess we could but that brings other problems, for example if some docker images are already preloaded on that disk image.

[zvin]

Perhaps a solution might be to add a "shrinking" step after successful preloading

Yes, but we still need a better estimation of the required space.

[dfunckt]

It would, this PR still stands though.

Okay, as long as you pad the estimate generously enough (currently 40%, right?) so we don't run out of space and fail, I'm with you.

[pdcastro]

[...] can you not create temporary space and copy the layers over when done?

I guess we could but that brings other problems, for example if some docker images are already preloaded on that disk image.

What about cloning the original image file as a "draft image", enlarging it with oversized estimations, pulling the images there, computing how much space was actually needed, then deleting the image and preloading the original one with the right size?

Probably no better than resize2fs. :-)

[zvin]

What about cloning the original image file as a "draft image", enlarging it with oversized estimations, pulling the images there, computing how much space was actually needed, then deleting the image and preloading the original one with the right size?

I think shrinking after preload would be simpler and faster (no copies).

@shaunmulligan do we still need to support btrfs ? Is anyone still preloading 1.x images ?

[pdcastro]

@zvin, just checking, are you waiting from Shaun's answer re btrfs support and/or intending to make further changes to this PR before it can be approved and merged?

[zvin]

@pdcastro No, I'm asking about btrfs for shrinking after preloading which should be in a separate pull request.