How to reconcile tutorials here with the notebooks in the tutorials repository

[ctrueden]

I am developing a (planned to be) comprehensive set of ImageJ tutorials as (Beaker for now, but hopefully later Jupyter too) notebooks, in the imagej/tutorials repository (on the dev-manual branch as of this writing, but that's soon to change). It is awesome to see more tutorials being developed and added here as script templates, but I am wondering what our general strategy should be for reconciling these two resources. Personally, I would like it if everything were somehow unified.

Maybe all the notebooks could be converted to scripts too and committed here? Or probably better, we could add a feature to the Script Editor to open Beaker/Jupyter notebooks as plain scripts directly? And add them as URL resources to the menus? In that way, there would be no potential for "skew" and all notebooks could be made automatically available from inside ImageJ. The hitch is that this notebook-to-script conversion would need to be quite smart about @ parameters. When we get @hadim's SciJava Jupyter kernel going, they could be baked in to the notebooks directly; otherwise, it is more complicated unfortunately.

@hadim @bnorthan @imagejan What do you guys think? Should we just give up and keep maintaining these two separate sets of things? Or do you see a way they could work together?

[axtimwalde]

@ctrueden sorry for the off=topic, but is anybody working on Java tutorials? I.e. how do I make and deploy and ImageJ2 plugin? How do I make and deploy an Op?

[ctrueden]

@axtimwalde There are already Java tutorials now in the imagej/tutorials repository. However, I am going to delete most of them in favor of the notebooks, because we have found that 20 different Maven projects, each one illustrating a tiny little idea, and in no particular order, was overwhelming for new non-programmer types.

Instead, we now have this:

• https://github.com/imagej/example-imagej-command – A sample ImageJ Command plugin
• https://github.com/imagej/example-legacy-plugin – A sample ImageJ 1.x PlugIn

The intent being to use the above projects as a starting point for your own Maven projects.

I expect we will add a couple more as needed, such as:

• https://github.com/scifio/example-scifio-format (does not yet exist)

If there are other gaps you feel need to be filled specifically with Maven projects, of course we can create more example-<foo>-<bar>. But mostly, if you start from example-imagej-command you can adapt it to whatever sorts of ImageJ plugin(s) you want.

[bnorthan]

I am happy with a unified approach. I've used Jupyter Notebooks a lot, however I haven't had a chance to look at the new ImageJ Beaker notebooks yet. I would be fine with using Beaker Notebook as the main means of writing tutorials.

I may have more to add after I get a chance to look at how the Beaker Notebooks work.

[ctrueden]

@bnorthan Cool. It should become easier in coming weeks, since: A) I'll be merging stuff to master of imagej/tutorials; and B) I'll be autoconverting Beaker to Jupyter at least for display purposes on a jupyter branch, while we continue to sort out the best way forward. That way people can browse the rendered notebooks on GitHub.

I'd be interested in any concrete ideas you have about the logistics of unification here. As I explained above, I have some ideas, but I also see potential pitfalls/flaws. The requirements we need to meet are:

• easy for new people to approach and learn—I think numbered notebooks are great for this
• testable automatically, so that we know as soon as things break (as @hadim has pointed out)
• accessible from inside ImageJ, especially for the purposes of: A) illustrating @ parameters; and B) serving as starting points / templates for people's own scripts
• applicable to Maven projects inside IDEs—I don't think this one is a big problem since Java programmers can copy/paste code from a script and Java-ize it.

Some of these requirements are partially in conflict: e.g., a long notebook is nice for teaching in an ordered way, but probably poor as an easy-to-adapt starting template for your own scripts. There are probably solutions to many of these conflicts—e.g., in the case of the long notebooks, each subsection of a notebook could be split into its own template script, if done carefully/intelligently.

[hadim]

I think it's a good idea to unify notebooks and scripts but from my point of view notebooks and scripts don't share the same purpose sometime.

A systematic conversion between scripts <-> notebooks is maybe not ideal since some notebooks are mostly used to teach for example.

Why not having one repo with separated scripts and notebooks ? We could then have a testing framework for both and developers/contributors can choose to share code using scripts or notebooks in that repo.

Scripts will still be readable from Github and from Fiji (in the Script Editor). While notebook will be readable from Github (for Jupyter notebook).

Two things we need to figure out :

• Find a way to convert automatically Beaker Notebooks <-> Jupyter Notebooks.
• Find a way to ship Notebooks inside Fiji : here maybe we don't want to do that since Notebook will be mainly used inside Jupyter/Beaker and the code will still be available from Github.

[imagejan]

I mostly agree with @hadim, it makes sense to unify some things.

I think it would be useful to have a limited selection of templates available in the script editor. By this I really mean templates, i.e. some code you can take as a starting point for developing your own script. The most obvious example might be the Process Folder script, currently in an IJ1 Macro and a Python version. We should identify common recurring tasks that would benefit from having such templates available.

For all other code that serves merely as example/tutorial, we should put these into notebooks that can ideally be read and executed online, without the need of installing anything. This is the long term goal for the Jupyter/Beaker notebooks, right?

[ctrueden]

Thanks @imagejan. I agree with your points.

When I have a chance (after NEUBIAS some time), I plan to try implementing a feature which enables the Script Editor to import notebooks. We could then include them in Fiji as remote URLs, sort of like The File ▶ Open Samples menu, but for tutorials rather than images.

I also cross-reference imagej/tutorials#30 here, which is the main issue where we are tracking the transition of ImageJ tutorials to notebook form. I am in the process of merging what we have so far to the master branch of imagej/tutorials.

[hadim]

With .bkr files being deprecated and the good progress made by the scijava-jupyter-kernel, this issue should be reviewed.

We don't need the systematic bkr <-> ipynb files conversion anymore (which is a very good thing !).

Do we really want to systematically convert scripts to/from notebooks ? I keep thinking that both could live together alongside with Java tutorials.

We could have one single repo that contains : Java tutorials, notebooks and scripts. All being tested in order to find any regression in the main libs (IJ, ImgLib2, etc).

Then ScriptEditor shows scripts in its menu. The only thing that needs to be developed is a way to import notebooks in Fiji on-the-fly.

Besides that, the only task would be the repo unification (and tests writing too).