when packing, the resulting OCI layer from the parts may have content that is already in the Ubuntu base

cjdcordeiro commented 1 year ago

Bug Description

See https://chat.charmhub.io/charmhub/pl/o5wxpb65ffbfzy7bcmi8kzftzy

To Reproduce

See https://chat.charmhub.io/charmhub/pl/o5wxpb65ffbfzy7bcmi8kzftzy

Environment

Nothing in special

rockcraft.yaml

See https://chat.charmhub.io/charmhub/pl/o5wxpb65ffbfzy7bcmi8kzftzy

Relevant log output

The build will be fine, but the resulting ROCK will be larger than necessary and fail the efficiency tests. Example: https://github.com/canonical/oci-factory/actions/runs/5940848690/job/16110358625

Proposal

We should, at some point before packing the ROCK, cross check the primed contents of the parts with what's already inside the base Ubuntu image. When those contents are equal, we simply don't include them in the final squashed layer that is going to be inserted into the ROCK.

Here are some possibilities to explore:

using a sort of "phantom layer" that contains the base Ubuntu image, such that the craft lifecycle considers the Ubuntu base when doing the conflict resolution it already does
just before packing the ROCK, add a new function to compare the primed content and the Ubuntu base's content

tigarmo commented 11 months ago

We had an internal discussion today about this and decided to address the issue with two changes:

1) Take the packages in the base layer into account

First, Rockcraft will look at both the packages installed in the base image layer and the packages installed in the build environment. Any packages that are in the base layer but not in the environment (if any) will be installed as build-packages.

Next, the list of packages installed in the base layer will be used by craft-parts when deciding the dependency cut-off for stage-packages. For example, if "libexpat" is a dependency of a package in stage-packages but it's already installed in the base layer (and in the build environment, because of the previous step), then it won't be included in the part's install files.

This change by itself is enough to address duplications of deb-related files (and will also let us fix dependency handling for classic snaps in Snapcraft), but it doesn't cover Chisel slices. It's less of an issue because Chisel slices are more often used with bare bases (which have no contents anyway), but regardless the second change addresses this:

2) Take the files in the base layer into account

After prime (aside: as a callback probably), Rockcraft will go over all primed files and prune/remove those already present in the base layer, provided they are identical (same contents & ownership). For chisel slices this is where we'll prune away the files from base slices that are already provided by the base layer, but this will also cover any other files that might have been added during the build (e.g. a direct download/extract in an override-build).

cjdcordeiro commented 11 months ago

Thanks for the update. I have some questions:

Any packages that are in the base layer but not in the environment (if any) will be installed as build-packages.

Why? I get the conservative approach, but in some cases, is that even possible? E.g. base: 24.04 and build-base: 22.04. Ultimately, it will also increase the build time.

Effectively, what this is saying is: "build-base is only allowed with base: bare"

Why isn't 2) enough? They seem redundant. 2) will effectively cover 1).

canonical / rockcraft