Add free space check for non-delta pulls

[dustymabe]

With the upgrade from f25 to f26 I can easily bump into the limit of the default OS settings in FAH especially if anything has written content into the root partition. I managed to fill up the FS on one of my openshift nodes and now can't access it anymore. I think the need for basic checks of the filesystem is clear. I believe dnf does this too.

Is it possible to determine how large our update will be and do an FS usage check before we grab content?

[dustymabe]

In retrospect, having this in Fedora 25 by the time we released Fedora 26 would have been really nice.

[cgwalters]

With static deltas, we can. Otherwise, an unfortunate side effect of the content-addressed deduplication is that storage between old and new is entangled and computing how much space is required is difficult; there is an OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES that we're not using right at the moment for FAH.

And right now rebasing from f25ah to f26ah doesn't have deltas set up - but that's something we should do.

[dustymabe]

With static deltas, we can. Otherwise, an unfortunate side effect of the content-addressed deduplication is that storage between old and new is entangled and computing how much space is required is difficult;

yeah I figured it was a little more complicated. Could we check things as we pull content and dump out if the freespace goes below 500 MiB or something?

there is an OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES that we're not using right at the moment for FAH.

would there be benefit to using it?

[cgwalters]

I don't think we can hardcode limits in libostree - among other things we need to consider flatpak and repo mirroring use cases. However, we can special case the system repo - Unix filesystems classically reserve 5% space for the superuser and the like, and we could honor that. Although hmm, it looks like XFS is different.

Certainly we could add a configurable option to libostree, then frob it to a specific value for e.g. the Fedora Atomic Host.

[dustymabe]

I don't think we can hardcode limits in libostree - among other things we need to consider flatpak and repo mirroring use cases. However, we can special case the system repo

I think it would be safe to say that almost any filesystem (atomc host, flatpak repo, ostree mirror) that you end up with less than 512MiB on would be safe to stop transferring files and notify the user. This value could be configurable as you say.

also there is still this question:

there is an OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES that we're not using right at the moment for FAH.

would there be benefit to using it?

[cgwalters]

As far as OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES...it's part of a long history that predates deltas. See this thread. We don't have anything that actually parses it beyond a test case. It's got bugs; in a quick look it doesn't handle --link-checkout-speedup, i.e. we would need to change it to walk the whole commit.

Lately I've been thinking about adding a new concept: basically a tarball of the metadata objects for each commit. This would be a middle ground between pure archive fetches and deltas; IME the cost of metadata fetches is large, plus it hampers async. I'm not really sure how worth it it'd be though. Well, I mean if it was implemented magically it'd likely be worth using, but the maintenance/complexity of a 3rd thing in this space is really nontrivial.

Yet another wildly different approach would be to implement jigdo like functionality - we'd basically fetch the RPMs and unpack them, and record differences in a final file. That's got its own whole set of advantages and disadvantages.

Anyways I think the above free space check is going to be good enough to close this for now.