【Pipeline Enhancement】More Generalized Pipeline

[xp1632]

• It looks like path 2 in Possible Enhancement is in demand

• To achieve the general and batch in the pipeline

• We need to modify the current pipeline

---

Difference in Pipeline v1 and v2: https://github.com/Max-ChenFei/VPE_IP/issues/78#issuecomment-2027343270

---

Pipeline Feasibility for Python Library

• since we found out the hard part of v2 is actually java-python type, let's see the feasibility for a python library that requires no type conversion across different programming languages https://github.com/Max-ChenFei/VPE_IP/issues/83

[xp1632]

---

we can see, to batch or not to batch, depending on the code snippet, let's see if offering a code template that would vary based on different type of Ops would work

[xp1632]

How JIPipe do: getting input-output information from OpInfo and run it in java module

And i checked the JIPipe code again, only find they are justing calling Op based on scijava Module, nothing special

but they carefully defined all types of inputs and outputs:

[xp1632]

Difference between JIPipe and our methods

• JIPipe is still calling ImageJ functions in a Java environment
• Thus it just import the type information of inputs and outputs from OpService
• Initialize the inputs and outputs into the java module
• And run the scijava module : https://javadoc.scijava.org/SciJava/org/scijava/module/class-use/ModuleService.html

---

• For us, firstly in Python we don't have this scijava module

  • we need to run ImageJ op via PyImageJ in this way:

  • 

• besides, for some complex datatype in java, for example, HyperSphereShape, which is a subclass of Shape, we don't have an equivalent type in python

• 

• We have to import this java class with jimport

• and use it in a specific way: HyperSphereShape(5)

• which makes it hard for us to call all ops in a unified way

[xp1632]

Challenge type problems : step 1, java-python rough mapping

• Then how should we solve this problem?
• we solved the first step of this type problem by a basic mapping table:

• we defined a rough mapping relationship between input java type and the python datatype in VP project

• In this case, net.imglib2.algorithm.neighborhood.Shape input type provided by filter.median plugin's OpInfo is roughly represented as int, which is 5 when we use it as HyperSphereShape(range)

• range is 5, and HyperSphereShape(range) is defined by us in the code snippet

[xp1632]

Challenge type problems: step 2 and solution: more detailed mapping table

• Just as JIPipe defined all datatype for ImageJ2, to make it fit better into its own slot

---

• What we could do is to provide a more detailed version of mapping table: like this :
• during java -> python type mapping, we also provide the subclass we should import and the format of correct use it

---

• In this way, we could at least, saving the manual jimport in code snippet

[xp1632]

• [x] Step 0: We wrap up the pipeline into one class Pipeline in commit f794facaaa244c368d947d4007cdc0e31176b621

• [x] Step 1-->2: We add the missing support for simple op without jargs as described in https://github.com/Max-ChenFei/VPE_IP/issues/75#issue-2179252228

• [ ] Step 3: We try to also generate the code snippet based on detailed mapping table as in https://github.com/Max-ChenFei/VPE_IP/issues/80#issuecomment-2027409753
• [x] Detailed Mapping Table #90

  • [ ] Code Snippet Generation #88

• [ ] Step 4: We generate a particle analysis workflow in https://py.imagej.net/en/latest/Puncta-Segmentation.html where all nodes are not manual-added

[xp1632]

Step 1 ---->Step 2 from One Node filter.median to Second Node threshold.otsu

Problem 1 : Multi Operation list for one Op_name

• The initial pipeline is tested with filter. median
• While this plugin only returns one operation from OpInfo when we search by its Op_name

---

• Other plugins such as threshold. otsu may return a list of operations under the same Op_name

---

Solution 1: Op Matching based on Parameter Number and default priority:

• As we detailed explained here: https://github.com/Max-ChenFei/VPE_IP/issues/72#issuecomment-1981361593

• We referred to ImageJ's own Op matching:

• 

• And Solution is:

• Because by then, we can not get type information from static code snippet

• However, while we are batching the pipeline, and the code snippet itself would be generated by us

• We could consider other solutions

---

Solution 2: Multi-Node for Multi-Operation

• Now we are not generating One Node but Batched Node

• So actually we don't have to pick from the Operation List

• We can create all of them and let the user choose which operation node they want to use

• We can distinguish the operations under same Op_name by adding hints in nodes or give them Node1, Node2, Node3, etc

[xp1632]

Problem 2: Should we modify extraction rule of the extract_op_parameter method?

• At our first glance of code snippet for running threshold.otsu :

• we thought, oh in extract_op_parameter() , we only wrote the extraction rule to extract op parameter in

ij.op().run("filter.median", ij.py.jargs( ** Extracted parameters **))

- And for calling "threshold.otsu", we didn't use jargs(), should we add another extraction rule for it?

---

The answer is No!

• Let's check the jargs() method definition in PyImageJ:

• You may see jargs() is just call to_java() method for every arguments inside

• Then let's check the code snippet of calling threshold.otsu again

• we also used the jargs() in this case to_java for the input parameter process_image and turn it into jimage

---

Conclusion:

• There's only one unified way of calling ops via PyimageJ
• which is ij.op().run("op_name", ij.py.jargs( ** Input parameters **))
• We thought it is not unified, is just we are confused by the link code between each op calling
• link code is not what we should provide, we only provide op_running, the prep link code should be provided with other link node, not inside of each op node

[xp1632]

The key to get a more generalized pipeline

is to generate the Code Snippet

• However, we met some problems that not a more detailed mapping table could solve
• I've summarized the code snippet generation in #88

---

We've finished the initial code snippet generation in https://github.com/xp1632/VPE_IP/commit/a99d0e08458d529fd7f74f6a3f42367bb7354c32

• However, two problems left to be solved :

  • how to deal with Multi-operation under one op name #91
  • Tuple related processing code adding in Format module in JSON generation

[xp1632]

Multi-operation under one op name problem

• This problem is not only a code snippet generation scope problem, but also how the node spec generation handle a one op name(filter.median), multi operations under(operation 1, operation2)
• I'll summary the details in #91

[xp1632]

Demanding JSON structure for ImaegJ nodes

• We want our ImageJ nodes look like this in VP project's search menu:

• 

• The file structure layer should be

  • ImageJ folder
    • namespace such as filter as a subfolder
    • all filter_xxx_xxx nodes in one JSON file

---

To achieve this kind of structure, we need a file structure as :

• and define the package in NodeTypePackage:

---

• This should be achieved when we change the structure of two methods to suit one op_name--> multi Op method demand

[xp1632]

Generate Structured JSON files for ImageJ package:

• We've finished the initial version of generation pipeline for ImageJ library in commit 01fc242d647acd8cfd6b7ed9043854ecb94f09db and issue https://github.com/xp1632/VPE_IP/issues/91#issuecomment-2074862790

• Demo video: https://drive.google.com/file/d/1haC6Uzsn63hzrRMwIq1kLHsBSrUdFZXu/view?usp=drive_link

• We generate node specifications that are grouped by its namespace:

• 

• We try to install the whole ImageJ_Package in NodeExtension's FilePond:

• but it somehow didn't work for whole package

---

Reason could be

• that we lack __init__.json for the package
• that we need to add this package in the NodeTypePackage as Kornia do in here
• I'll figure it out next week

---

Reason for previous not-working package import

• Yes, we lack __init__.json for the package
• after adding it:

• 

• we get our desired file structure after a one-time package import instead of several json import

• 

[xp1632]

Thoughts about incompatible tuple, list type in Chaldene

• I don't think it's a good idea to make specific modification in generation pipeline for the type tuple, list input that Chaldene as a visual programming language doesn't support for now
• This is not a part of enhancement for pipeline but rather a remedy for Chaldene
• So I think this is where we finalize the pipeline after adding the __init__.json to achieve package import