RFC: Decouple Dependencies from the Buildpacks

[ForestEckhardt]

Readable

[loewenstein]

I wasn't able to grasp the current PR with change suggestions any longer and maybe you are right @ForestEckhardt and some of my comments go beyond review and towards an alternative proposal. So I started sketching that in https://github.com/paketo-buildpacks/rfcs/blob/decouple-dependencies-alternative/text/0000-decouple-dependencies.md based on the branch used in here.

Happy to hear what you think of it.

cc @dmikusa

[dmikusa]

The proposal is intentionally abstract. How metadata and actual binaries become available to the buildpacks running in the container is meant to be open-ended so that we can support multiple platforms and use cases. I fully realize that makes it harder to follow.

I agree with @ForestEckhardt in that what we really need to specify is the metadata format, both the directory/file structure as well as the structure inside of the metadata files. The nebulous processes that run outside of the container to provide the metadata need to meet and coordinate with the buildpack code through this interface.

I'm simplifying a lot here, but I think most other things can be worked out later or through follow-up RFCs. Things like where the binaries live. That's really a concern between a particular buildpack and a particular metadata provider (or possibly between language family teams). If patterns eventually emerge, we can turn them into RFCs so we have consistent behavior.

For example. We could consider pack and possibly some manual work (or a tool to be defined later) to be the external metadata provider. It has created the prescribed directory structure and metadata files. It then volume mounts that into the container at the prescribed location. The buildpack knows where to find those files because of this RFC and how to read them.

I think what happens next is out of scope for this RFC, because otherwise, it's going to make this RFC really, really big and hard to implement and it's already pretty large and complicated.

There are a number of questions though, carrying on from the example, what happens next? Presumably, the buildpack will install some dependencies, but what dependencies does it install? What version/how does it select a version? What dependency formats does it support (tar/rpm/deb/etc..)? What transports does it support? etc...

I'd like to leave this type of stuff out of this RFC, so that we can get the interface together first. Then we can experiment with things and see what ends up being useful and common across Paketo buildpacks. For example, maybe down the road we decide that we want a generic Paketo buildpack that can install any requested dependency by an id or name from local file systems/HTTP and understands tar archives, and uses semver. To me, that's hard to know right now and is big enough to make its own RFC.

[loewenstein]

We do not have any nebulous tool creating dependency metadata yet though but will rather develop them as part of this endeavor, don't we?

Hence, I believe this RFC is indeed too specific. I would prefer to first agree on "yes, we see the potential benefits of the decoupling and should pursue this" and then actually figure out the details while workings on it.

I'd start with a minimal set, one buildpack per buildpack library in use in the Paketo project and probably a human as the first nebulous tool to provide metadata and assets to the buildpack.

[dmikusa]

We do not have any nebulous tool creating dependency metadata yet though but will rather develop them as part of this endeavor, don't we?

Nothing yet. From the Paketo standpoint, I'd expect to have tooling in this area for CI, similar to jam and other tools we have now that manage dependencies.

For other use cases, I don't really expect Pakto to provide those. Maybe we look at providing a tool for local dev, since that's a common use case? Maybe we let projects like Spring that implement buildpacks integration provide that?

Hence, I believe this RFC is indeed too specific. I would prefer to first agree on "yes, we see the potential benefits of the decoupling and should pursue this" and then actually figure out the details while workings on it.

I definitely don't want this to be overly specific, but at the same time, I don't think we need an RFC just to agree on pursuing this effort. I'd like to think that by now someone would have voiced a counter-opinion if it is not a thing we should pursue. So I'm hoping we can have an RFC that gives us some minimal foundation on which we can start building things.

I'd start with a minimal set, one buildpack per buildpack library in use in the Paketo project and probably a human as the first nebulous tool to provide metadata and assets to the buildpack.

Not sure I follow you here. Sorry.

[loewenstein]

Alright, I do believe that the lifecycle of metadata and assets, if assets are to be provided, should be coupled not separated.

I do believe that dependency bundles (the mix of metadata and potentially assets of some set of dependencies) should scale from small to large. I.e. a user should easily be able to provide a single dependency as well as the language family maintainers should be able to provide larger ones. The former would benefit from a single metadata file, while the latter could be achieved with e.g. allowing a file system structure to easily add and group namespaces.

If the same dependencies come from different dependency bundles, we need a way to figure out precedence.

I would however like to avoid putting these rough thoughts into formal specification just yet, but rather align on goals to reach.

[loewenstein]

I'd start with a minimal set, one buildpack per buildpack library in use in the Paketo project and probably a human as the first nebulous tool to provide metadata and assets to the buildpack.

Not sure I follow you here. Sorry.

What I meant was that we have all things under control, so there is little need to specify details upfront instead of just diving into it and discover the specifics we need as part of the journey.

[dmikusa]

@loewenstein @ForestEckhardt - Let's move this discussion to Slack, and get a little more consensus. I think we're expecting too much out of GH Issues here to keep everything in order :) I've invited you both to a channel. It's public, so others interested are welcome, I just didn't want to spam everyone with invites so I only started with us.

[ForestEckhardt]

This RFC has been drafted to indicate that a component of it if being worked on in another RFC and that this is not my main focus. This RFC will be undrafted when it is the main focus again.