Feature request: new integration point before the build

[orpiske]

Hello,

I have been investigating the feasibility of using JBang to propose an alternative solution for running java code snippets for our projects. If successful, it would be possible to simplify how we execute them.

The current support for integration [1] allows me to implement most of necessary bits to achieve that. However, there is one item that I could not solve with it: injecting dependencies before the build actually happens.

The reason to manipulate the list of dependencies before the build is because the modelines for JBang and the project I am working on are different.

Whereas JBang uses:

//DEPS log4j:log4j:1.2.17

The project uses something like:

// my-project: dependency=mvn:org.my:application:1.0

Because I would like to keep the current support for the existing modeline, I would need to be able to parse them prior to the JBang build wrapper, in order to inject these dependencies.

Do you think it would be possible to include a new integration point (i.e.: preBuild) for that?

1. https://www.jbang.dev/documentation/guide/latest/integration.html

[maxandersen]

interesting idea but it has issues...like if this integration point can resolve dependencies then ordering and final resolution set is not going to be deterministic.

//DEPS log4j:log4j:1.2.17
// my-project: dependency=mvn:org.my:application:1.0
//DEPS some:xyz:1.24

what would the order be ?

or are you suggesting that you can transform the content of the file of the file in memory before jbang parses so it will see //DEPS org.my:application:1.0 instead?

[quintesse]

That to me seems the easiest solutions at least. Otherwise we're indeed going to go down a rabbit hole. Although perhaps it could be just after parsing but before we do anything with the parse result. Meaning that the integration could poke around in the //-lines before we actually resolve them.

[orpiske]

or are you suggesting that you can transform the content of the file of the file in memory before jbang parses so it will see //DEPS org.my:application:1.0 instead?

I haven't thought about this possibility originally, but I think that would work as well.

The main goal is to be able to transform from one format to another so that I can make jbang see //DEPS log4j:log4j:1.2.17 when the user has written //my-project: dependency=mvn:log4j:log4j:1.2.17. So, anything that can trick it into that - without requiring our users to manually modify their code - would be cool.

[maxandersen]

so I was about to try enable this when I hit the first snag/blocker.

This is while parsing for dependencies so no way for dependencies to be loaded to be called to do something to the // lines.

Not seeing a good way to fix that unfortunately - do you?

[maxandersen]

what project is this for btw?

what is the full syntax/semantics you need covered ?

could consider adding it as alternate syntax like we support @Grab syntax in additon to //DEPS.

[orpiske]

what project is this for btw?

We are evaluating this as a way to run the Camel K local mode.

what is the full syntax/semantics you need covered ?

I'll summarize below, but for the sake of reference, our dependency resolution mechanism is described in these 2 pages:

• https://camel.apache.org/camel-k/latest/configuration/dependencies.html
• https://camel.apache.org/camel-k/latest/cli/modeline.html

The most important bit is the one defined on the modeline. We let our use define explicit dependency using a modeline like:

// camel-k: dependency=mvn:org.my:application:1.0

There's also some automatic dependency resolution happening under the hood, but it's not really important in this case because our Java DSL supports handles it (and it works well w/ the current integration mechanism from JBang).

could consider adding it as alternate syntax like we support @grab syntax in additon to //DEPS.

I think that's certainly an alternative we could consider. What I am trying to do with this is to create a proof of concept for Camel K's local mode that would avoid requiring the Golang-based CLI (aka Kamel). There's some interest on the community for that and the idea is to propose this to the community for when the work on 2.0 starts. However, it would prevent the new CLI version from being backwards compatible. I think being backwards compatible would make a much greater proposition for the community to consider the idea.

[maxandersen]

so we could probably find a way to support "mvn:*" but the part of shortcuts that has some magic camel lookup mechanism i.e. "camel-mina2" thats not going to fly if it needs some dynamic list resolution?

[quintesse]

Another (more complex) option would be to have integration dependencies. Eg. have something like //INTDEPS which are resolved first and can be used to pull in integrations.

But of course that would mean that you'd still need to add some special/extra lines to your source files.

Yet another option might be that the integration dependencies could be passed on the command line using the --deps option. But that would make the invocation by the user more complex, so also not an ideal solution.

A third option could be that integrations are things that a user "installs" and which are then applied to any files that get built. The current Quarkus integration could be one that is pre-installed while the Camel one could be one that a user has to install once. Eg. jbang integration install camelk@camel.

[orpiske]

so we could probably find a way to support "mvn:*" but the part of shortcuts that has some magic camel lookup mechanism i.e. "camel-mina2" thats not going to fly if it needs some dynamic list resolution?

Yes, that's one of the things it would need to convert. I was looking at the pre-work [1] I had I prepared*, and my notes mention that it would need to be converted somehow.

1. https://github.com/orpiske/camel/blob/jbang-integration/dsl/camel-jbang/camel-jbang-runtime/examples/Kafka2Jms.java#L18-L33

A third option could be that integrations are things that a user "installs" and which are then applied to any files that get built. The current Quarkus integration could be one that is pre-installed while the Camel one could be one that a user has to install once. Eg. jbang integration install camelk@camel.

That sounds interesting. Not just for this Camel K proposal, but for bringing other runtimes that may rely on different modelines or more complex transformation of the code before it gets fed into javac via JBang.

[maxandersen]

so one concern is that jbang is not "free" to call - it does take precious startup time thus we are heading towards or at least considering making it a native app; meaning both current and future integrations preferably are done by external process tooling rather than dynamically loaded or linked java libs. It would be more like a integraiton that gets given the content of a .java file and allows to process it it file by file.

[maxandersen]

And after writing this I kinda just realized #1000 could be a first usecase :)

''' jbang app install jbang-camel@camel jbang camel youkamelk.java '''

Then camel is simply a script that preprocesses the files to make them jbang "compatible" and then call Jbang run on the result.

Just a thought.

[quintesse]

Indeed! That is a much simpler solution that won't suddenly affect all source files passed to JBang but will only be used when specifically run.

[orpiske]

And after writing this I kinda just realized #1000 could be a first usecase :)

''' jbang app install jbang-camel@camel jbang camel youkamelk.java '''

Then camel is simply a script that preprocesses the files to make them jbang "compatible" and then call Jbang run on the result.

Just a thought.

I think that would work.

[davsclaus]

btw camel-jbang is now able to parse those modeline files and dynamic add dependencies on our own - there is also other dependencies that Camel detects during bootstrap that would not be possible to determine (without a lot of pain) without starting Camel first.

[maxandersen]

And you do that by basically wrapping jbang in a camel-jbang script, right?

[davsclaus]

And you do that by basically wrapping jbang in a camel-jbang script, right?

yes we start jbang the same way with camel-jbang (it has a few DEPS only) and then we at runtime compute which additional DEPS we need to run this camel "code", and then self download these inside camel.

We use the same approach as jbang with using maven apis for downloading ( we also used shrinkwrap previously )