Nextflow kernel

[pditommaso]

I'm interesting to contribute a kernel for Nextflow, that's a DSL/interpreter for scientific computational workflows running on top of JVM/Groovy.

In the case you are interested to such extension, is there anybody that could advice how to implement it? Being mainly a Groovy application, I guess it would require little more than repacking the Groovy kernel, adding the Nextflow libraries.

[scottdraves]

Hi Paolo that looks cool, happy to help. It might be even possible to just dynamically load it into the Groovy kernel, depending on its embedding. Can you say more about how it works?

[pditommaso]

Do you mean, just loading the application jar on-fly. Maybe it's an option, however since NF applies come complex AST transformations on Groovy code to implement its own DSL, I guess the a better option is to implement a specific kernel.

Also the main goal of the integration would be to execute long running workflows. Therefore it's required to provide some kind of visual feedback that's not supposed to be the immediate result of the evaluation of a few lines of code.

[scottdraves]

OK in that case, yea you'll need your own kernel and you can inherit from our base kernel, or maybe from the groovy kernel. You should be able to add a custom progress widget, our spark integration does this for example.

We are going to break up BeakerX into multiple packages so you can depend on just the part you need. I don't want to include it into our repository since I can't maintain it, but I can help you make it a seamless experience for your users.

[pditommaso]

We are going to break up BeakerX into multiple packages so you can depend on just the part you need

Nice, any idea when this is going to happen?

I don't want to include it into our repository since I can't maintain it, but I can help you make it a seamless experience for your users.

I agree. Can you highlight the main steps to move it forward? I mean: 1) Which is supposed to the entry entry point? 2) What base class to extend 3) How to setup a development environment to test the new kernel?

[scottdraves]

Not sure what you mean but https://github.com/twosigma/beakerx/tree/master/kernel/groovy/src/main/java/com/twosigma/beakerx/groovy/evaluator has the relevant code. It might be possible to depend on the Groovy kernel to make a kernel. The normal architecture, they all inherit from the base kernel: https://github.com/twosigma/beakerx/tree/master/kernel/base

The README should cover how to get it compiling.

[scottdraves]

Another example: https://github.com/scijava/scijava-jupyter-kernel

[altavir]

I understand the answer, but I would also like to state my own intentions. I intend to use beakerx as a front-end for DataForge and therefore implement appropriate kernel. For now I plan to use kotlin kernel (or groovy if something goes wrong with kotlin, because kotlin shell is not very mature), but in future, I will probably need specific one. I don not currently have time to develop it, but any documentation on how to add your own kernel would be welcome.

[pditommaso]

Not sure the last comment is related to Nextflow kernel that's the topic of this issue?

[altavir]

I am just saying that guidelines for new kernels are needed not only for Nextflow. There will be more.