Adds Squash Stacks RFC - Githubissues

ryanmoran commented 2 years ago

Summary

Readable

Checklist

[x] I have viewed, signed, and submitted the Contributor License Agreement.
[ ] I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
[ ] I have added an integration test, if necessary.
[x] I have reviewed the styleguide for guidance on my code quality.
[x] I'm happy with the commit history on this PR (I have rebased/squashed as needed).

dmikusa commented 2 years ago

💯 This will be fantastic!

fg-j commented 2 years ago

I was curious how many layers this would save (i.e. how many layers are in each current stack) - it seems like each stack currently contains ~7 layers. So, while it won't permanently solve our problems regarding builders with too many layers, it'll create some wiggle room for people creating custom builders that add to the Paketo Full builders.

dmikusa commented 2 years ago

I was curious how many layers this would save (i.e. how many layers are in each current stack) - it seems like each stack currently contains ~7 layers. So, while it won't permanently solve our problems regarding builders with too many layers, it'll create some wiggle room for people creating custom builders that add to the Paketo Full builders.

Hmm, interesting. I think I misunderstood this RFC. Is this to just squash the run/build image layers? or is it to squash down the builder layers? or both?

I agree, squashing the run/build images isn't going to save a ton, but squashing the builder layers, that would solve the layer problem for us.

fg-j commented 2 years ago

Perhaps a discussion of squashing builders should be added here. Right now, we create builders with pack builder create, which doesn't have a --squash option. A potentially more reusable implementation here is to add a jam squash-image command that flattens an arbitrary image. This could be used for both builders and stacks.

fg-j commented 2 years ago

Something to bear in mind with potentially squashing builders: If builders squash all buildpacks into a single layer, whenever any buildpack in the builder changes, the entire cached layer will be invalidated. This will significantly hurt builder-pull performance on any platform that uses on OCI layer caching. Maybe it would be more prudent to squash composite buildpacks before distributing them in builders.

dmikusa commented 2 years ago

Something to bear in mind with potentially squashing builders: If builders squash all buildpacks into a single layer, whenever any buildpack in the builder changes, the entire cached layer will be invalidated. This will significantly hurt builder-pull performance on any platform that uses on OCI layer caching. Maybe it would be more prudent to squash composite buildpacks before distributing them in builders.

That's a fair point. I think it would be interesting to try both and see how it goes. I'd be curious to see how big the layer ends up being with all the buildpacks squashed into one layer. There are quite a few individual layers now with the builder image and that's not exactly speedy. Sometimes downloading one larger resource is faster than downloading many smaller resources.

fg-j commented 2 years ago

All in all, if this RFC introduces a more general jam squash-image subcommand and sets the intent to (at least) squash stacks, we can experiment and decide on a best builder-squashing approach after implementation. I'm all for experimenting on this.

ryanmoran commented 2 years ago

I don't think we'd likely want a builder squashed into a single layer. More likely, we'd want to keep the base stack layer, and then squash each of the buildpack groups into a layer. It feels like it'll be harder to do that from the outside.

davidmirror-ops commented 2 years ago

WG 11-15-2022: [RM] think about a more intelligent way to squash layers [FGJ] What about generalizing this, and implementing it as a CLI command? [RM] API design would be needed

fg-j commented 2 years ago

After looking a bit crane flatten, I'm fine with this proposal as-is. Separately, I think it'd be worthwhile to explore using crane flatten on language family buildpacks before we publish them.

ryanmoran commented 2 years ago

@paketo-buildpacks/stacks-maintainers I'm looking for any more feedback on this.

sophiewigmore commented 2 years ago

I'm fine with proceeding with this as-is. We can extend support further for builders, etc in the future if we want to.

paketo-buildpacks / rfcs

Adds Squash Stacks RFC #260

Summary

Checklist