Independence from ImageJ and ImageJ2; support for Icy and Matlab

[haesleinhuepf]

A long-term wish from some users was to craft a jar file that is independent from ImageJ and/or ImageJ2. We should implement this later but have it in mind from the very beginning. I think this could be achieved by having packages structured like the following. We can later split these packages into separate repositories with different dependencies.

• net.clesperanto.core: This package contains all the functionality for processing images (e.g. gaussian_blur. A user could theoretically run such code:

  
  from net.clesperanto.core import Clesperanto as cle

cle.gaussian_blur(...)

However, this gateway does not contain `push()`, `pull()` and `imshow` functions. It might be marked as purely internal package.

* `net.clesperanto.imagej`: This package contains things that depend on classical ImageJ such as `ImagePlus` and `ImageStack`. Users could run such code:

from net.clesperanto.imagej import Clesperanto as cle

gpu_image = cle.push(imageplus) gpu_result = cle.gaussian_blur(gpu_image, ...) imageplus = cle.pull(gpu_result)

imageplus.show() IJ.run(imageplus, ...)

* `net.clesperanto.imagej2`: This package contains things that depend on ImgLib2 and ImageJ2 such as `RandomAccessibleInterval`, `IterableInterval` and `DataSet`. Users could theoretically run such code:

from net.clesperanto.imagej2 import Clesperanto as cle

gpu_image = cle.push(imageplus) gpu_result = cle.gaussian_blur(gpu_image, ...) rai = cle.pull(gpu_result)

ij = new net.imagej.ImageJ() ij.ui().show(rai)

* `net.clesperanto.icy`: This package contains things that depend on Icy, such as  [`push()`](https://github.com/clij/clicy/blob/5f423ee2c70b1567cfeec00f936a525ef678f377/src/main/java/net/haesleinhuepf/clicy/CLICY.java#L49) that takes a sequence as input. 
* `net.clesperanto.matlab`: This package contains things that depend on Matlab-specific code, such as [MOCL](https://github.com/clij/clatlab/blob/master/src/main/java/net/haesleinhuepf/clatlab/MOCL.java)

If we strive for such a solution, we would have multiple `Clesperanto` classes and their `push()` and `pull()` functions would have different parameter and return types. 

Alternative: In CLIJ we had a `push()` function retrieving any type and multiple `pull()` functions such as `pullRAI()` for ImageJ2, `pullMat()` for Matlab and `pullSequence()` for Icy compatibilty.

Just for completeness, I'm adding the python version
* `pyclesperanto`: This package is naming-wise different to follow best-practices in pypi/python ecosystem. Users could run such code:

from pyclesperanto import cle

gpu_image = cle.push(numpy_array) gpu_result = cle.gaussian_blur(gpu_image, ...) numpy_result = cle.pull(gpu_result)

This will not be implemented very soon. Opinions welcome!

[tinevez]

Hello @carlosuc3m , @haesleinhuepf , all. If I may here are a few suggestions.

A long-term wish from some users was to craft a jar file that is independent from ImageJ and/or ImageJ2. We should implement this later but have it in mind from the very beginning. I think this could be achieved by having packages structured like the following. We can later split these packages into separate repositories with different dependencies.

Yes this is a great idea. Actually it might becomes a requirement in the future with the future version of JDK that require artifact separation to be matching package declarations :/ But anyway it's an elegant pattern and we strive for elegance.

• net.clesperanto.core: This package contains all the functionality for processing images (e.g. gaussian_blur. A user could theoretically run such code:

from net.clesperanto.core import Clesperanto as cle

cle.gaussian_blur(...)

However, this gateway does not contain push(), pull() and imshow functions. It might be marked as purely internal package.

Ok with this we would achieve independence from the data structure underneath 'images', whatever they are. And I see the importance of this if we want to simplify our lives.

I would instead make the core specific to interfaces, with a specific data structure, and I would pick imglib2. We can make the core inly depends on imglib2 interfaces, no concrete implementations.

It's a compromise, because now we have something specific to imglib2. But the nice thing is that imglib2 is basically everywhere.

• the cle-Fiji would be 2 classes.
• the same for cle-Icy.
• the cle.ImageJ2 would not be needed at all (it is based on imglib2 interfaces already)

For MATLAB I am not sure yet. We can go through Java with the imglib2 bridge, but we could also make a C-binding? I would be in favor of addressing the first 3.

[tinevez]

Suggestion for package names

---

CORE

net.clesperanto.jclic

Everything CLIC:

• uses float[] and int[] to exchange data
• device managements
• kernel calls Nothing more. Only java.lang imports. Example classes in this package: ⁃ ArrayJ ⁃ Tier1, etc

---

CORE with imglib2.

net.clesperanto.jclic.imglib2

Limit the uses of imglib2 to this package (nothing above, so that we can ship the artefact without imglib2 dip). Contain a class that maps the native calls to imglib2 interfaces.

[tinevez]

To manage and facilitate the copy from / to array[] / imglib2 interfaces:

https://github.com/imglib/imglib2/pull/330

[carlosuc3m]

Hello, I am already working on this and I wanted to give you a small update so we can see if I am going on the right direction and also to ask a couple of questions about the design.

Core Inside src/main there are now 3 packages:

• net.clesperanto.presets (required for Javacpp)
• net.clesperanto.core (contains ArrayJ, MemoryJ, DeviceJ, BackendJ)

• net.clesperanto.imglib2 (Methods to convert imglib2 into ArrayJ and vice-versa)

  For the core, I am basically wrapping the src/gen net.clesperanto.wrapper.clesperantoj.... into new classes. This makes the code more understandable for end users but also might be confusing. The names of the classes are the same and if their IDE has some code suggestion it might be tricky at first. As you can see below for every class in the core there will also be a suggestion for the class autogenerated by JavaCPP:

With respect to the actual processing routines in each Tier. My understanding is that we need to do the same with a code generation Script or will the users call for example Tier1.absolute(....) from net.clesperanto.wrapper.kernelJ?

ArrayJ Currently ArrayJs can only be copied from the GPU into the CPU, but I think it would be interesting to be also capable of referencing the byte array from the GPU. In this way ArrayJs could be directly modified. Also ImgLib2 arrays can be created directly from a byte buffer: https://github.com/clEsperanto/clesperantoj_prototype/blob/fd511d3698fefb44fcb31b21d6813624da5ea4ad/src/main/java/net/clesperanto/imglib2/Converters.java#L52

Thus we could reference directly ArrayJs from ImgLib2 arrays for visualization or single pixel access.

It would also improve the conversion of ImgLib2 arrays into ArrayJ as it currently needs two copies. I am using the ImgLib2 Blocks API as suggested by @tinevez. In this process you need first to create a float array of the size that you want to copy and then copy it to the GPU, thus 3 copies (the ImgLib2 array, the float array with the values, and the GPU array). I have limited experience with C++, so some feedback from @StRigaud would be appreciated, but I assume that in order to reference the byte array from the GPU we would need to expose a new method in the clesperantoJ native code. I can try to do it myself if @StRigaud thinks this is possible.

ArrayJ-ImgLib2 The ArrayJ- ImgLib2 conversion is in another package that is never used within the core, making the ImgLib2 dep optional as we wanted.

https://github.com/clEsperanto/clesperantoj_prototype/blob/fd511d3698fefb44fcb31b21d6813624da5ea4ad/src/main/java/net/clesperanto/imglib2/Converters.java#L27

This class contains static methods to copy from RandomAccessibleInterval to ArrayJ and from ArrayJ to ImgLib2 ArrayImg (it will be extended to other ImgLib2 Imgs). Right now it only supports Float conversion but I will extend it now to every data type.

There are just two public static methods that convert back and forth: https://github.com/clEsperanto/clesperantoj_prototype/blob/fd511d3698fefb44fcb31b21d6813624da5ea4ad/src/main/java/net/clesperanto/imglib2/Converters.java#L43

https://github.com/clEsperanto/clesperantoj_prototype/blob/fd511d3698fefb44fcb31b21d6813624da5ea4ad/src/main/java/net/clesperanto/imglib2/Converters.java#L77

Here is where I would also like to have a method that creates an imglib2 referencing the byte array of an ArrayJ.

Sorry for the long comment but I wanted to make sure that I am following the correct direction.

Regards, Carlos

[carlosuc3m]

Also, I wanted to ask another couple of things. How the conversion from nd-array to flat array should be done, Fortran-order or C-order? ImgLib2 does fortran order but numpy does C-order.

In which order are the bytes stores on the GPU, Little or Big endian?

[StRigaud]

For the core, I am basically wrapping the src/gen net.clesperanto.wrapper.clesperantoj.... into new classes. This makes the code more understandable for end users but also might be confusing.

Couldn't it be done by simply renaming the output generated by JavaCPP? tell JavaCPP to directly generate a net.clesperanto.clesperantoj ? Or do we have to pass by an extra layer?

Currently ArrayJs can only be copied from the GPU into the CPU, but I think it would be interesting to be also capable of referencing the byte array from the GPU. In this way ArrayJs could be directly modified.

The GPU memory bytearray can only be accessed by OpenCL operation. What is stored in Array is a pointer to the memory address on the GPU that we get when create the data, but this GPU memory is not the same as the CPU memory, we have to rely on the OpenCL API hidden in the C++ code. Accessing the bytearray on GPU is done through the methods write and read of Array which, for now, only provide full memory read but can also access sub-part of the memory, the methods are just not visible to Java yet.

What we need is to access a pointer array from an imglib2 and give it to the write or read method of the ArrayJ without have to copy the content of the imglib2 into a new pointer array and then pass it to the ArrayJ method (which is what we will do if no other option)

How the conversion from nd-array to flat array should be done, Fortran-order or C-order?

OCL store the way you code it, if I didn't messed-up we are in c-style

[carlosuc3m]

Couldn't it be done by simply renaming the output generated by JavaCPP? tell JavaCPP to directly generate a net.clesperanto.clesperantoj ? Or do we have to pass by an extra layer?

yes it can be done directly simply by changing the output of the java cpp code, as done here: https://github.com/clEsperanto/clesperantoj_prototype/commit/3d5608d5e6f599aa1298b9831d095391e5edc3dd

In my experience however I have always found working with code directly generated by JavaCPP a little bit challenging.

The GPU memory bytearray can only be accessed by OpenCL operation. What is stored in Array is a pointer to the memory address on the GPU that we get when create the data, but this GPU memory is not the same as the CPU memory, we have to rely on the OpenCL API hidden in the C++ code. Accessing the bytearray on GPU is done through the methods write and read of Array which, for now, only provide full memory read but can also access sub-part of the memory, the methods are just not visible to Java yet.

What we need is to access a pointer array from an imglib2 and give it to the write or read method of the ArrayJ without have to copy the content of the imglib2 into a new pointer array and then pass it to the ArrayJ method (which is what we will do if no other option)

Great now I understand better

OCL store the way you code it, if I didn't messed-up we are in c-style

okk

[StRigaud]

In my experience however I have always found working with code directly generated by JavaCPP a little bit challenging.

got it. Normally I would not mind but we are already using wrapper on the C++ style with the native code which wrap the reel C++ code CLIc. If we add another Java wrapping layer it become maybe too much Idk.

It's ok for me if it does not make maintenance too complex or impare speed.

[carlosuc3m]

got it. Normally I would not mind but we are already using wrapper on the C++ style with the native code which wrap the reel C++ code CLIc

makes a lot of sense.

The wrapper is done already so I can show you in the next meeting and we can decide.

For maintenance, I think we would need a Script that generated the wrapper for the functions if we were to decide to keep the wrapper.

[carlosuc3m]

EDIT: the snippet of code provided only works with memory type: buffer, it does not work with image.

Here there is some progress on the core-imglib2 relationship.

The code below opens a file with ImageJ, converts the ImageJ ImagePlus into an ImgLib2 Img, sends the data to the GPU and does Gaussian blurr along y-axes.

It seems to work well. The ImgLib2 <-> ArrayJ conversion is done in this isolated class: https://github.com/clEsperanto/clesperantoj_prototype/blob/reorg-carlos/src/main/java/net/clesperanto/imglib2/Converters.java

This snippet of code uses the wrappers for the JavaCPP generated code, which are up for discussion. Removing them would be straightforward and require minimal effort.

    public < T extends NumericType< T > & NativeType< T > > Example1a()
    {
        // open a file with ImageJ
        File file = new File( "wing_float.tif" );
        final ImagePlus imp = new Opener().openImage( file.getAbsolutePath() );

        // wrap it into an ImgLib image (no copying)
        Img<T> image = ImageJFunctions.wrap(imp);
        // display it via ImgLib using ImageJ
        ImageJFunctions.show( image );

        BackendJ.setBackend("opencl");
        DeviceJ currentDevice = new DeviceJ();
        System.out.println(currentDevice.getInfo());

        ArrayJ arrayj = Converters.copyImgLib2ToArrayJ(image, currentDevice, "buffer");
        ArrayJ arrayj_out = MemoryJ.makeFloatBuffer(currentDevice, image.dimensionsAsLongArray(), "buffer");

        Tier1.gaussianBlur(currentDevice.getRaw(), arrayj.getRaw(), arrayj_out.getRaw(), 1, 30, 1);
        ImageJFunctions.show( (RandomAccessibleInterval<T>) Converters.copyArrayJToImgLib2(arrayj_out) );
    }

This is the pom file needed to run this small example:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.scijava</groupId>
        <artifactId>pom-scijava</artifactId>
        <version>37.0.0</version>
        <relativePath />
    </parent>

    <groupId>net.imglib2</groupId>
    <artifactId>imglib2-tutorials</artifactId>
    <version>0.1.0-SNAPSHOT</version>

    <name>ImgLib2 Tutorials</name>
    <description>This project shows ten increasingly complex examples of how to program with ImgLib2. The intention of these examples are not to explain ImgLib2 concepts, but rather to give some practical hints how to work with the library and to grasp the principles in a learning-by-doing way. Also included is the Game of Death (similar to Conway's Game of Life): a cellular automaton, simulating competing life forms, which illustrates the generality of the ImgLib2 typing mechanism.</description>
    <url>https://imagej.net/ImgLib2_Examples</url>
    <inceptionYear>2009</inceptionYear>
    <organization>
        <name>ImgLib2</name>
        <url>http://imglib2.net/</url>
    </organization>
    <licenses>
        <license>
            <name>Simplified BSD License</name>
            <distribution>repo</distribution>
        </license>
    </licenses>

    <developers>
        <developer>
            <id>tpietzsch</id>
            <name>Tobias Pietzsch</name>
            <url>https://imagej.net/User:Pietzsch</url>
            <roles>
                <role>lead</role>
                <role>developer</role>
                <role>debugger</role>
                <role>reviewer</role>
                <role>support</role>
                <role>maintainer</role>
            </roles>
        </developer>
        <developer>
            <id>StephanPreibisch</id>
            <name>Stephan Preibisch</name>
            <url>https://imagej.net/User:StephanP</url>
            <roles>
                <role>founder</role>
                <role>lead</role>
                <role>developer</role>
                <role>debugger</role>
                <role>reviewer</role>
                <role>support</role>
                <role>maintainer</role>
            </roles>
        </developer>
        <developer>
            <id>axtimwalde</id>
            <name>Stephan Saalfeld</name>
            <url>https://imagej.net/User:Saalfeld</url>
            <roles>
                <role>founder</role>
                <role>lead</role>
                <role>developer</role>
                <role>debugger</role>
                <role>reviewer</role>
                <role>support</role>
                <role>maintainer</role>
            </roles>
        </developer>
        <developer>
            <id>ctrueden</id>
            <name>Curtis Rueden</name>
            <url>https://imagej.net/User:Rueden</url>
            <roles>
                <role>maintainer</role>
            </roles>
        </developer>
    </developers>
    <contributors>
        <contributor>
            <name>TODO</name>
        </contributor>
    </contributors>

    <mailingLists>
        <mailingList>
            <name>Image.sc Forum</name>
            <archive>https://forum.image.sc/tags/imglib2</archive>
        </mailingList>
    </mailingLists>

    <scm>
        <connection>scm:git:https://github.com/imglib/imglib2-tutorials</connection>
        <developerConnection>scm:git:git@github.com:imglib/imglib2-tutorials</developerConnection>
        <tag>HEAD</tag>
        <url>https://github.com/imglib/imglib2-tutorials</url>
    </scm>
    <issueManagement>
        <system>GitHub</system>
        <url>https://github.com/imglib/imglib-tutorials/issues</url>
    </issueManagement>
    <ciManagement>
        <system>GitHub Actions</system>
        <url>https://github.com/imglib/imglib2-tutorials/actions</url>
    </ciManagement>

    <properties>
        <package-name>net.imglib2.tutorials</package-name>

        <license.licenseName>bsd_2</license.licenseName>
        <license.projectName>ImgLib2: a general-purpose, multidimensional image processing library.</license.projectName>
        <license.organizationName>ImgLib2 authors</license.organizationName>
        <license.copyrightOwners>Tobias Pietzsch, Stephan Preibisch, Stephan Saalfeld,
John Bogovic, Albert Cardona, Barry DeZonia, Christian Dietz, Jan Funke,
Aivar Grislis, Jonathan Hale, Grant Harris, Stefan Helfrich, Mark Hiner,
Martin Horn, Steffen Jaensch, Lee Kamentsky, Larry Lindsey, Melissa Linkert,
Mark Longair, Brian Northan, Nick Perry, Curtis Rueden, Johannes Schindelin,
Jean-Yves Tinevez and Michael Zinsmaier.</license.copyrightOwners>
    </properties>

    <repositories>
        <!-- NB: for SciJava dependencies -->
        <repository>
            <id>scijava.public</id>
            <url>https://maven.scijava.org/content/groups/public</url>
        </repository>
    </repositories>

    <dependencies>
        <!-- ImgLib2 dependencies -->
        <dependency>
            <groupId>net.imglib2</groupId>
            <artifactId>imglib2</artifactId>
            <version>7.0.2</version>
        </dependency>
        <dependency>
            <groupId>net.clesperanto</groupId>
            <artifactId>clesperantoj_</artifactId>
            <version>0.0.0.1</version>
        </dependency>
        <dependency>
            <groupId>net.imglib2</groupId>
            <artifactId>imglib2-ij</artifactId>
        </dependency>
        <dependency>
            <groupId>net.imagej</groupId>
            <artifactId>ij</artifactId>
        </dependency>
    </dependencies>

</project>

I am using the images from the imglib2 tutorial: https://github.com/imglib/imglib2-tutorials/tree/master

[carlosuc3m]

As discussed last week with @StRigaud we are trying to decide how we should present the different Tier methods in clesperantoj. Currently we are creating wrappers on the Javacpp generated code as, in my opinion, it is not clear at all. I have come up with a series of prototype wrappers for the clic methods that are defined and explained in this pull request: https://github.com/clEsperanto/clesperantoj_prototype/pull/40

The key point is that there exist the possibility of accepting ImagePlus, ImgLib2 RandomAccessibleIntervals or Icy Sequences Objects in clesperantoJ without compromising the standalone use of the library without dependencies. This is also explained on the pull request (https://github.com/clEsperanto/clesperantoj_prototype/pull/40)

CC @tinevez @haesleinhuepf

[carlosuc3m]

Just some update after @StRigaud and I discussed the possible prototypes. We have reduced the number of options to just 1:

• public static ArrayJ absolute(DeviceJ device, Object input, ArrayJ output)

Where the input object can be ArrayJ, RandomAccessibleInterval, ImagePlus..., the ouput parameter can be null or and existing ArrayJ and the value returned will always be an ArrayJ.

The mehtod is intended to be used like this:

    // parameter input can be imglib2 or imageplus or arrayj
    // parameter output can be arrayj or null
    // return type is arrayj (or other specific type)
    // and we can use it as follow:
    // 1- ArrayJ output = absolute(dev, input, null);
    // 2- absolute(dev, input, output);

The pull request is still the same: https://github.com/clEsperanto/clesperantoj_prototype/pull/40