Proposal to create language family builders

[kevin-ortega]

Readable

[ryanmoran]

Today we offer 3 different "flavors" of builder based on the stack they align with. How would we deliver a coherent set of language-specific builders as we offer more stacks (or "flavors") going forward?

For instance, what does it mean to have a builder called paketobuildpacks/builder:nodejs or paketobuildpacks/builder:go? Would the Node.js builder build on top of the base Jammy stack while the Go builder built off the tiny stack? And what of other stacks? Where does this leave UBI or a future Alpine stack?

[kevin-ortega]

Today we offer 3 different "flavors" of builder based on the stack they align with. How would we deliver a coherent set of language-specific builders as we offer more stacks (or "flavors") going forward?

For instance, what does it mean to have a builder called paketobuildpacks/builder:nodejs or paketobuildpacks/builder:go? Would the Node.js builder build on top of the base Jammy stack while the Go builder built off the tiny stack? And what of other stacks? Where does this leave UBI or a future Alpine stack?

I'm updating the RFC on how to deal with jammy, ubi and future ubuntu-based stacks. I'm not familiar enough with the other language families and the full, base, tiny flavors and what each family would need. Is it sufficient for the RFC to suggest each language family will need to choose what stack their respective builder should be based on?

[ryanmoran]

I think I can get onboard with the current language about the variants that would be available for each language family.

I also see that the builder.toml example provided composes the component buildpacks that make up the current Java buildpack. Is the expectation that we would get rid of the language-family buildpacks and replace them with builders. My concern with this is that we end up with significant duplication, or even incidental skew in the buildpacks included in each builder. It also prevents someone from easily using a "buildpackless" builder and the language-family buildpack to build.

What are the benefits of pushing the language-family order configurations into the builders?

[sophiewigmore]

WG notes: Looking for feedback from maintainers of other language families and builders

[kevin-ortega]

I think I can get onboard with the current language about the variants that would be available for each language family.

I also see that the builder.toml example provided composes the component buildpacks that make up the current Java buildpack. Is the expectation that we would get rid of the language-family buildpacks and replace them with builders. My concern with this is that we end up with significant duplication, or even incidental skew in the buildpacks included in each builder. It also prevents someone from easily using a "buildpackless" builder and the language-family buildpack to build.

What are the benefits of pushing the language-family order configurations into the builders?

I don't propose to replace language-family buildpacks with builders. I will update the sample builder.toml to reflect that.

[johnnyr0x]

@ryanmoran to follow up with team to get more input

[dmikusa]

@paketo-buildpacks/builders-maintainers - Would you be able to weigh in on this one? Primarily, the ownership is going to be on the language-family team, but language-family teams will use builder tools so there is some shared ownership. Thanks!

[ForestEckhardt]

I think the job of owning and maintaining builder tools makes sense to fall under the guise of builder maintainers. My only quest is if the generic buildpacks will continue to exist in this system and if they do would that not force the need for upstream intervention anyway?

I think if the argument is that language specific builders would be good to have regardless of the layer limit that is a fair assessment but it we are only doing this to avoid the layer limit I think that I would rather we lobby harder upstream.

This is a big chunk of work and I just want to be secure in the intentions.

[kevin-ortega]

I think the job of owning and maintaining builder tools makes sense to fall under the guise of builder maintainers. My only quest is if the generic buildpacks will continue to exist in this system and if they do would that not force the need for upstream intervention anyway?

I think if the argument is that language specific builders would be good to have regardless of the layer limit that is a fair assessment but it we are only doing this to avoid the layer limit I think that I would rather we lobby harder upstream.

This is a big chunk of work and I just want to be secure in the intentions.

Certainly the layer limit is one driving force for this proposal. Equally important, however, is the performance improvement (faster downloads) and reduced size gained by having family language builders. The generic builders will continue to exist with this proposal.

[dmikusa]

+1 to what @kevin-ortega said. I think there's still value for this beyond the layer limit. I would also add that as the Java ecosystem of buildpacks grows, it probably doesn't make sense to put everything into the generic builders. If we see the intent of the generic builders as an easy way to get started, then we want to put an opinionated set of buildpacks in there that supports most users out of the box. The Java language family builder can then be a complete and total set of everything, all the JVM providers, all the app servers, all the APM vendors, etc...

[kevin-ortega]

This looks good to me.

The only thing I'd call out is that OSS support for Bionic will be dropped in April 2023 - both by Paketo but also by Canonical - so by the time we approve and implement this RFC there may not be much value in implementing a Bionic builder for language families. Therefore we may wish to remove it from the RFC. I don't mind if we keep it in the RFC though in the interests of getting it approved sooner.

The ownership detailed in the Implementation section seems fine, with the addition that some of the machinery for the builders is co-maintained with @paketo-buildpacks/tooling-maintainers (i.e. the builder section of the shared github config)

Good point. I have updated the RFC and removed bionic examples.

[sophiewigmore]

Working group 3/15 - waiting on a final review from @ForestEckhardt

[ForestEckhardt]

My last question after looking over this is where would the repo for the builder.toml and automation be? Would they be added as sub directories to the existing builder repo? Would we create new language specific repos mono repos? Would we create a language specific repo per stack?

I don't see that laid out in the RFC and would like to have an idea before final approval.

[dmikusa]

My last question after looking over this is where would the repo for the builder.toml and automation be? Would they be added as sub directories to the existing builder repo? Would we create new language specific repos mono repos? Would we create a language specific repo per stack?

I don't see that laid out in the RFC and would like to have an idea before final approval.

Good question. I agree that we should make a decision on this point now.

My thought was that we'd have a language-specific repo. The existing builder repo seems logically focused on that builder, so I don't think that makes sense to add language-specific builders there.

As far as mono repo or multiple repos, I think it's up to the tooling. If it supports a mono repo that would be my preference since it seems like less work to maintain. If the tooling doesn't support that then it would be multiple repos.

I will say I don't have strong feelings about this though and I would happily defer to @paketo-buildpacks/builders-maintainers since they have experience maintaining the existing builders.

[robdimsdale]

I think it will be easier for the builders team to maintain if we have separate repos per builder. So one repo for builder-jammy-full-java, one for builder-jammy-tiny-java, etc. The existing tooling should work fairly well with minimal overhead for that structure. If we want to adopt a monorepo approach we'll have to spend additional time and effort adapting the existing automation to work with that. Not an inherent problem - just pointing it out.

[ForestEckhardt]

I am fine with separate repos from a builder maintainer perspective but that does increase the number of repo for participating language families by a quite a bit. Would just want to make sure that we are comfortable with adding that level of overhead.

[robdimsdale]

This comes down to the monorepo vs separate repo debate.

I personally don't feel that a monorepo is significantly easier to maintain when we're talking about objects that are a relatively decoupled - like builders. I'd feel differently if there were lots of shared dependencies that had to be kept in sync - I just don't think that describes this situation.

That being said, I'm open to working on builder automation for the multiple-builder-per-repo pattern. It just won't work out of the box the way it would for separate builder-per-repo.

[dmikusa]

I'm 👍 with that. I'd rather stick with the way things are now, so we can get this going. If that proves to create problems, then we can discuss down the road how to address those.

[robdimsdale]

Great, so in terms of naming, the current (jammy) naming convention is as follows: builder-<OS>-<variant> e.g. builder-jammy-full.

For the language-specific subtypes, I'd propose adding the subtype as a suffix (rather than somewhere else like in the middle or as a prefix). So that would look like: builder-jammy-full-java or builder-jammy-base-java-with-extras.

[kevin-ortega]

Great, so in terms of naming, the current (jammy) naming convention is as follows: builder-<OS>-<variant> e.g. builder-jammy-full.

For the language-specific subtypes, I'd propose adding the subtype as a suffix (rather than somewhere else like in the middle or as a prefix). So that would look like: builder-jammy-full-java or builder-jammy-base-java-with-extras.

I agree and I will update the RFC with this naming convention.

[dmikusa]

Thanks for the comments and votes. I think we have the votes to proceed, so this is the final chance to comment.

If there are no additional comments or objections by EOD Friday, Mar 24th, then we'll go ahead and merge this. Thanks!