Investigate Spring Buildpack dependency for potential Re-Use.

[BarDweller]

The Spring Boot Maven & Gradle Plugins are able to create Docker images using Buildpacks.

It looks like they have isolated their buildpack logic to a common dependency, which is independently published to Maven

Investigate if it's possible for other Java projects to consume that dependency to create container images via Buildpacks.

[cmoulliard]

This feature is very interesting as it could interest the projects Dekorate/Quarkus.

• What do you think abut this feature also for Dekorate @iocanel ?
• Should we contact Clément/Taco of Quarkus to discuss such an idea ? @geoand

[iocanel]

+1 for dekorate support.

[cmoulliard]

Is there a buildpack API or spec that we could rely on to develop the Java Buildpack Lib ? @BarDweller

[BarDweller]

Looks like there are docs here.. https://buildpacks.io/docs/reference/

[geoand]

• Should we contact Clément/Taco of Quarkus to discuss such an idea ? @geoand

I think it would be better if we first had something to present - like a PoC on Quarkus

[BarDweller]

Initial Results:

Spent a while reading through the Spring 'spring-boot-buildpack-platform' dependency, and built a simple driver app that uses it to perform a buildpack build. In under 10 lines, it's possible to have it run the buildpack and create the application image.

Sounds good, but digging deeper I become less convinced.

Some aspects of the dependencies API are package protected, notably, you can't implement their custom log receiver as one of the methods accepts an argument of a type that's declared with package visibility. (Easy to workaround by placing your log impl into the same package as that class, but it just feels wrong).

It's probably fair to suggest that the API for the dependency is unpublished, and likely considered internal by Spring Boot, not in a 'you cannot use this way', as Apache license says 'go use this' ;), but more in a 'This API is not stable, and we are still figuring out what it does, and doesn't do'. Eg, I spot that in SB 2.5, they will be adding the ability to push the image directly to a registry, and there are various changes between the 2.3 and master branches that are sufficient enough to warrant some caution against consuming the API as a dependency.

That much said, it's almost a shame it's embedded with the Spring universe, as it does appear to be an almost complete minimal implementation of the buildpack platform using Java. SB 2.3 implements Buildpack 0.3, their master branch is tracking Buildpack 0.4 (Buildpack is currently at 0.5). From what I can tell following the code, and inspecting the images produced, they are compliant to the spec.

The spring buildpack impl is mostly (90%) a java docker client, the parts that orchestrate the build mostly being limited to the o.s.b.b.p.build package, which represent a thin abstraction over the spec concepts. Most of the buildpack spec centers around pulling the right images, and executing the spec mandated commands in the context of the appropriate image, with appropriate arguments set. https://github.com/buildpacks/spec/blob/buildpack/0.5/platform.md Spring choses the simple path of executing /cnb/lifecycle/creator .. https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-tools/spring-boot-buildpack-platform/src/main/java/org/springframework/boot/buildpack/platform/build/Lifecycle.java#L123-L147 You can see it there passing the args to the creator binary, each arg is documented in the spec against the individual lifecyle phase that owns it. Note that most of the args are Directories bound into the container to let the build pack do its job.

It's also worth noting the Spring buildpack dependency brings in a whole pile of other deps, some are obvious, eg, jackson is present to parse / manage the json blobs that sit within the image metadata used by buildpack images, other libraries there for talking to docker. But it also brings in spring-core (which brings in spring-jcl) .. seemingly just to use Spring Asserts and other Spring util classes. Together the base spring deps add 1.5mb to the buildpack libs 179 or so kb.

So I think in conclusion, if we wanted to reuse the Spring dep as a buildback lib, we'd need to work with them to ensure decent API interface (that didn't rely on package protected classes as part of its externals), and be willing to tolerate the transient dependencies of Spring-Core/JCL as part of the cost of consuming it for other uses.

If either of those are deal-breakers, then we're basically down to creating our own Java Buildpack library. We might be able to consume an existing Java Docker library, although we may find we need to contribute upstream changes depending on the level of access to metadata that such libraries already expose etc. Consideration will likely need to be made for podman/buildah etc as docker alternatives, if they do not have existing Java APIs that would need to be handled first.

[cmoulliard]

From what I can tell following the code, and inspecting the images produced, they are compliant to the spec.

At least this is a good news

[cmoulliard]

So I think in conclusion, if we wanted to reuse the Spring dep as a buildback lib, we'd need to work with them to ensure decent API interface (that didn't rely on package protected classes as part of its externals), and be willing to tolerate the transient dependencies of Spring-Core/JCL as part of the cost of consuming it for other uses.

Can you then open a ticket and ask to spring boot engineers what they think about your idea/proposition ? @BarDweller

[cmoulliard]

If either of those are deal-breakers, then we're basically down to creating our own Java Buildpack library.

I suspect that the snowdrop team will be more than happy to develop it then ;-)

[cmoulliard]

We might be able to consume an existing Java Docker library, although we may find we need to contribute upstream changes depending on the level of access to metadata that such libraries already expose etc.

Can you provide your lights concerning a Java Docker or CRI-O library that we could re-use here ? @iocanel

[cmoulliard]

But it also brings in spring-core (which brings in spring-jcl) .. seemingly just to use Spring Asserts and other Spring util classes

Do you think that Quarkus will accept if we create a quarkus buildpack extension that we have such a dependency with spring project ? @geoand

[cmoulliard]

I think it would be better if we first had something to present - like a PoC on Quarkus

Maybe but first we should first know what they have started to do, design, @geoand ...

[BarDweller]

Can you then open a ticket and ask to spring boot engineers what they think about your idea/proposition ? @BarDweller

I know one of the Devs on the project from past Spring work, I had a quick chat with him on Slack this morning.. and he had this to say re the buildpack dependency.

We've considered extracting that code into a separate lib. the whole CNB landscape was changing pretty fast this whole year, with builder and lifecycle API changes etc. it's better for Boot to keep it internal for now so we can iterate as quickly as possible. we might revisit that as things settle down.

So sounds like it may be safer to build our own, at least for now.

[geoand]

But it also brings in spring-core (which brings in spring-jcl) .. seemingly just to use Spring Asserts and other Spring util classes

Do you think that Quarkus will accept if we create a quarkus buildpack extension that we have such a dependency with spring project ? @geoand

The extensions doesn't even need to be in Quarkus core (probably won't be), but in the https://github.com/quarkiverse (that's where more non critical extensions are meant to reside these days).

[geoand]

I think it would be better if we first had something to present - like a PoC on Quarkus

Maybe but first we should first know what they have started to do, design, @geoand ...

I don't know of anyone that has done in buildpack related work in Quarkus

[cmoulliard]

I don't know of anyone that has done in buildpack related work in Quarkus

Maybe I got the wrong information from Max. Can you please check that with him ? @geoand

[geoand]

Sure yeah

[cmoulliard]

The extensions doesn't even need to be in Quarkus core (probably won't be), but in the https://github.com/quarkiverse (that's where more non critical extensions are meant to reside these days).

Should the build be owned by Quarkus ? I know that Quarkus docker extension is doing this job but I'm not sure that we should package such an image build process within a framework instead of delegating this responsibility to another tool/system able to perform different builds ...

[geoand]

I don't have an opinion on how it should be implemented. I'm just saying that we can now do anything we want with an extension as the extension doesn't need to be in the core.

On Tue, Nov 3, 2020, 19:40 Charles Moulliard notifications@github.com wrote:

The extensions doesn't even need to be in Quarkus core (probably won't be), but in the https://github.com/quarkiverse (that's where more non critical extensions are meant to reside these days).

Should the build be owned by Quarkus ? I know that Quarkus docker extension is doing this job but I'm not sure that we should package such an image build process within a framework instead of delegating this responsibility to another tool/system able to perform different builds ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/snowdrop/team/issues/296#issuecomment-721277549, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBMDP5PM37T7ZUAT3OSATDSOA57DANCNFSM4THUDOZA .

[cmoulliard]

Do you plan to impl the Java Lib according to this API spec - https://github.com/buildpacks/spec/blob/buildpack/v0.5/buildpack.md ? @BarDweller

[BarDweller]

Do you plan to impl the Java Lib according to this API spec - https://github.com/buildpacks/spec/blob/buildpack/v0.5/buildpack.md ? @BarDweller

We'd follow the platform spec.. https://github.com/buildpacks/spec/blob/main/platform.md

The buildpack spec is for buildpacks themselves, platform spec is for tools that use buildpacks. So we'd need to follow platform spec to create something that drive buildpack builds.. and if we also create buildpacks for quarkus, those would follow the buildpack spec.

[geoand]

I just read the entire description by @BarDweller and it's great information!

It sounds to me like we really don't want to depend on something so tied to Spring. Would it make sense for us to have our own implementation and if so how difficult would it be to achieve do you think?

[BarDweller]

I think at this stage, I agree, we probably want to code our own, Spring's impl will be useful at least to confirm understanding of the spec =)

As for how difficult? if we assume we can use an existing java docker api implementation, then the remaining code looks relatively containable.. I'll have a quick bash at prototyping before I'm willing to give any sizing that can be relied upon ;) from what I know so far, it feels like a few weeks effort assuming an existing docker impl. If it turns out the existing docker impls out there are unsuitable, that could easily triple, although I suspect it may be simpler to pick the best candidate for a docker impl, and work to extend that to become what we need. My concerns here are mostly related to how buildbacks are storing data in the image metadata, and how accessible/manipulable that data is via existing apis.

Also if we're building our own, we'll need to be careful re requirements gathering to understand what we need from such a library. I'm already aware we'll want to make sure it works with podman (as well as docker). But thoughts on how to handle loggers, what level of customisation (for buildpack selection, base/run image selection, caches, image naming etc) will be expected. I'm aware of what the Spring lib has surfaced, but it's possible we don't need all of that, just as we may want to expose other features. As a prototype, I'll start by building something 'similar' in scope, just because that'll help me firm up my understanding of the spec.

[BarDweller]

Update: I've finished digging and I'm going to start coding up a proof of concept,

I wanted to at least uncover why Spring chose the path of creating their own Docker client, I was guessing that either the standard docker java client was insufficient, or that they just didn't want the whole thing. It's definitely the first, it would seem that the pack cli isn't just talking to docker, it's also handling stuff that users wouldn't normally do.

From all my reading of the code, I was missing one key part, "how does the resulting image end up in the docker repo?"

I could see that the actual build is executed by invoking a supplied binary in the context of a container formed from the build image, but given all that is happening 'in' docker, there had to be some magic occurring for the resulting output to end up back in the same docker =) The build pack creates the output image layers into a mounted dir, and the pack cli (or Spring java client) then assembles the layers into an image, that's imported via docker load. That image assembly is the missing part in the docker java clients, as it's not a normal docker client behavior.

Implementing the image assembly doesn't look too hard, although I'm sure there are a million edge cases to get right to remain compliant.. I still plan to use the docker java client as much as possible, that at least isolates us from docker api changes etc, and (subject to testing) should also work with podman (or maybe podman v2, I'm chatting to the podman team to make sure I don't do anything daft ;) )

I suggest we close this issue here, if we don't plan to consume the Spring dependency, and open a new Issue to track the PoC work.

[cmoulliard]

I suggest we close this issue here, if we don't plan to consume the Spring dependency, and open a new Issue to track the PoC work.

Instead of closing maybe rename the title and remove spring mentions ? Why: To have all the history in one ticket @BarDweller

[maxandersen]

I don't know of anyone that has done in buildpack related work in Quarkus

Maybe I got the wrong information from Max. Can you please check that with him ? @geoand

we do want to explore enabling buildpacks - the serverless work with quarkus is done via buildpacks and "dream" is we can combine tekton + buildpacks efforts and not duplicate.

having a local java only library to work with buildpacks and then wire that up nicely with our quarkus openshift/kubernetes story with a quarkus-container-buildpack and/or quarkus-openshift extensions would be sweet.

[cmoulliard]

we do want to explore enabling buildpacks - the serverless work with quarkus is done via buildpacks and "dream" is we can combine tekton + buildpacks efforts and not duplicate.

As soon as we will have this java buildpacks lib, then it will become very easy to create a Tekton task as this one https://github.com/tektoncd/catalog/blob/master/task/buildpacks/0.2/buildpacks.yaml#L85-L108 and to rely on dekorate tekton to generate the tekon manifests needed to launch a build on the kubernetes tekton cluster

[maxandersen]

Not following why this help on tekton support for buildpacks - they already have that don't they ? I see this as a way to build containers with buildpack locally without requiring installing anything but Java ?

[BarDweller]

I've put up where I've got to so far at https://github.com/BarDweller/java-buildpack-client

Overview: It's functional, and will build a working app image from source, or uber-jar. It uses the docker-java (https://github.com/docker-java/docker-java) project to handle docker low-level requests, and commons-compress to build the tarballs used to transfer content to docker. It requires docker, with a docker daemon, I'm about to start testing with podman etc. It does work fine on Windows, tested with my local Docker on WSL2 on Win10.

Technical info: The client mounts an appropriate bunch of volumes to act as cache dirs for the build, and copies the source into a container built from the buildpack 'buildImage'. The creator process is then driven within that container to carry out the entire buildpack build process. The docker socket is mounted into the build container, and the build uses that to publish the built image back to the local docker. As an alternative, the build can run without the daemon, but then it needs a docker registry to upload the final image to.

TODO: It's not quite spec compliant yet, as it doesn't validate the build/run image metadata, and it reads the whole app content into memory and back out when building the container (so if the app is too big, it'll hang today, it's on my list of things to fix). There's limited progress support (the build log is relayed, but docker pulls are silent atm),

Next Steps: We need to start collecting requirements =)

Requirements Questions: These need answers from people who might consume this client as a library...

• where do we need our target images to go ? local daemon vs remote registry / with/without auth ?
• do we have a stack we should be using instead of the paketo one? (buildImage/runImage pair)
• do we have custom buildpack content we want to use via this? (would be present in a spec formatted directory as an arg)
• do we want to pass custom env vars to the final image? (assume map string>string is fine?)
• how will application content be specified (File/Directory/Programmatically supplied as Path/size/inputstream per entry)
• what kind of log/feedback is required during execution?
• do we want the buildpack to assemble from a war/uber-jar, or let it build from source, and take care of everything?
• can we tell if a build is for 'the same application' (allowing us to reuse cache volumes), how can this be indicated to the builder?

[cmoulliard]

As an alternative, the build can run without the daemon, but then it needs a docker registry to upload the final image to.

This extension could be valuable as no docker daemon is then needed locally. That will of course slow down the build/push process

[cmoulliard]

• do we have a stack we should be using instead of the paketo one? (buildImage/runImage pair)

This is a question to be asked to the runtimes team or runtimes java team

[cmoulliard]

• where do we need our target images to go ? local daemon vs remote registry / with/without auth ?

Can we review what jib offers (= to compare the options they propose: https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin#build-your-image) first, next podman : http://docs.podman.io/en/latest/markdown/podman-build.1.html ...

[cmoulliard]

• do we want to pass custom env vars to the final image? (assume map string>string is fine?)

Should we pass such parameters to the image or instead rely on K8S Deployment/Container parameters ?

[BarDweller]

Issues tidy up.. closing this issue, in favor of the enclosing epic.

With this sub task considered as 'investigate spring buildpack dependency for reuse' has resulted in a decision to not reuse the spring dependency.

I'll bring the general requirements to a new task in the epic.

[maxandersen]

What is the "enclosing epic", got a link?

[BarDweller]

What is the "enclosing epic", got a link?

Via the Epics link in the top right of the issue (may require zenhub?) https://github.com/snowdrop/team/issues/319