Buildpack binaries using JBang?

[BarDweller]

Buildpacks require a 'build' and a 'detect' binary (or script) to perform the corresponding parts of the buildpacks responsibilities during a buildpack build.

Today these are written in bash, but they would be more maintainable, and more capable if we can elevate from bash to a higher language.

Paketo use Golang, and have a library to enable their model.

I propose using Java via JBang, as we likely have more Java coders around who can then maintain the resulting buildpack.

[BarDweller]

On investigation, the lifecycle invocations of the buildpack build/detect binaries is made by a Golang executable.

It would appear Golang is unable to directly invoke a jbang executable, it would appear the issue lies with the implementation of Go's exec call that does the os level fork. Even if we got a fix for that, we'd need to wait for a lifecycle release from buildpacks to be built using the fixed level of Go.

Instead, going to investigate use of a simple bash launcher that merely passes all args to a JBang invocation..

[BarDweller]

Investigations complete..

Detect/Build scripts for Buildpacks can be written in bash, or must exist as Binaries.

JBang unfortunately is a script, that doesn't start with #! this seems to confuse the process that invokes the Detect/Build process, which attempts to fork to the JBang script as if it were a Binary. Investigation showed the fault lies within the Golang os.exec call, so it's not something that would be fixed quickly.

Workaround is to drop in two wrapper scripts that in turn each invoke the JBang script. This works, and only has minimal impact. However, it doesn't appear possible (yet) to precache the built JBang script, I need to look into this a little more.. if it isn't possible, then we might want to just stick with bash & bash functions.

I had a look at trying to use Maven api's to determine the total dependency graph of a Maven project, to determine if any were dragging in io.quarkus, to enable us to distinguish a Quarkus Maven project from other Java Maven projects. Turns out this isn't simple at all, and looks like the easier route is to use the Maven invoker api to have it drive the dependency list goal, and then read back the result from a file. Work in Progress.

[maxandersen]

Would "jbang whatever.java" not work?

Btw. You Can replace // with !# in jbang script it's just that then that Line Will show up with a syntax error.

[BarDweller]

Dropping this idea, as it's looking like with the eventual evolution of buildpacks allowing for ubi8+deps via rpm, that it would be unlikely that any builder base image we use would have a functional jre/jvm to allow use of JBang.

As an alternative, we could consider using Quarkus Native, although the on-disk footprint is substantially larger than say Go, or Bash, it may still offer enough of an advantage to make it worthwhile, especially if we coded it as a single binary that selects it's function based on the symlink it was invoked via (ala busybox). UPX compression would be appropriate for it's inclusion in the builder image, where on disk size is likely of greater importance than brief memory overheads while executed.

It is however looking increasingly likely that we may contribute to Paketo, (using Go), in which case this would not be required.. (however a Quarkus Native binary to discover the jre/jdk version via maven/gradle etc may yet be worth it, but that's definitely worthy of a different task, if/when we reach that point.

[maxandersen]

Isn't quarkus native and jbang builds completely different usecases or rather orthogonal ?

Buildpack with jbang is to allow for super simple setup...which optionally could provide a quarkus binary.

[BarDweller]

Here, I was talking about the 'detect' and 'build' scripts/binaries that buildpacks use to perform the build.

This isn't about the language of the app being built, but rather about how a given buildpack performs that build.

Buildpacks (and extensions) use the 'detect' script to ask if a given buildpack wants to participate in a build, and the 'build' script to tell it to perform its job if selected. The buildpack documented examples are written as bash scripts, and our builder images that we have created to date also use bash scripts. Paketo on the other hand uses Go for their build/detect implementation, and have created a framework that's used by all their various buildpacks, for tasks such as installing a jre/jdk to building a dot.net application.

If we were to build our own buildpack ecosystem, distinct from Paketo, then once you look at the challenges posed by creating a wide variety of buildpacks, combined with them each having common elements (eg, parsing the buildplan.toml or outputting provides/requires correct), you most likely end up at the same conclusion that our current bash scripts will become unwieldy and we'd likely also want to create a framework of some sort.

Given that the invocation points today are either scripts or binaries.. JBang was an interesting possibility here, allowing us to leverage the Java skills we have available, but, it has 2 main drawbacks.

• GoLang cannot exec a JBang script, as it doesn't recognise the magic at the start. (GoLang is only using the system exec to invoke that, but JBang's use of // rather than #! is non standard and unlikely to be supported by the standard exec fn.) This point is relatively trivial to work around, just add a wrapper invocation script, and it's back to working..
  • The builder image will not have a JRE/JDK by default. This one is likely the showstopper.. the builder image needs to be as lightweight as possible, as it will be used to build apps for different runtimes, Go (and Quarkus Native) have the advantage of not requiring a JRE/JDK to execute their apps. JBang (afaik) will still require one. I don't think this one can be worked around, except by adding a JRE/JDK to the builder image, which would increase the size, and possibly introduce issues when a different level of JDK/JRE is request by the app..

If we become part of the Paketo ecosystem, and in some way adopt that by way of offering support, then we'll already have to be able to fix issues in the existing Go logic contained in their buildpacks, so it kind of makes sense to have anything we create also be part of that framework. (Indeed, the quarkus buildpack contribution to Paketo is already in Go).

There may be exceptions to this, the most likely one I'm aware of currently is 'how to determine the level of Java being requested by a maven project, without having Java installed' (and at a guess a similar issue exists also for gradle/sbt etc). Taking maven as a particular example here, the java level may be configured within the app pom.xml, or it could just as easily be set via a parent pom, or a parent-parent etc up the chain. Making this more complex, is that the parent pom may not be on disk locally, it could be in a repository configured via the pom. Correctly understanding this, is maven's job, attempting to reimplement that aspect in bash, or GoLang feels.. problematic.. especially edge cases, and tracking version compat etc.. This is where it may make sense not to use Go even if we are creating Paketo extensions, and maybe use a Quarkus native command line app that links the maven libs so it can query the project for property values. Although this may be a problem more easier solved by just saying 'we'll default to version X, and you can override it by setting this env var' (which is kind of what the other Paketo jre/jdk supplying buildpacks do today).