Open xp1632 opened 7 months ago
Knowledge Graph
and Display nodes
in the first stage, what contents should be in the first VP ImageJ workflow?Display nodes
, now we don't have a formal Display nodes
in VP project.
captureImageCode
and captureImageFunctions
in python generator
to show intermediate image result
of a node on its node panelij.py.show(java image object)
as a temporary measureij.py.show(java image object)
should be added within every VP node
that has image invovled2.2 For Converters in Knowledge Graph
, we consider the KG part as the next step, we do not want to include imageJ converters in current knowledge graph.
Thus we'll unify the input/output image Type as numpy.ndarray
And do the conversion inside of VP function node
via PyImageJ's from_java()
, to_java()
Also, in PyImageJ only IJ1 ImagePlus
and RandomAccessibleInterval
are converted to numpy, so we'll use ij.IJ.openImage()
to read the image as the correct form
- Images we get from
ij.io().open()
returnsImageJ2 Dataset
java object.- Image we get from
ij.IJ.openImage() legacy opener
returnsImageJ1 ImagePlus
java object
{
"description": "Image read and write functions from torchvision.io",
"enable": true,
"nodes": {
"read_image": {
"type": "read_image",
"category": "function",
"title": "read_image",
"tooltip": "Reads a JPEG or PNG image into a 3 dimensional RGB or grayscale Tensor. Optionally converts the image to the desired format. The values of the output tensor are uint8 in [0, 255].",
"externalImports": "from torchvision import io\nfrom torchvision.io import ImageReadMode",
"codeGenerator": "function code(inputs, outputs, node, generator) {\n if (inputs[2] === 'ImageReadMode.RGB')\n return `${outputs[1]} = io.read_image(${inputs[1]}, ${inputs[2]})\\n${outputs[1]} = {\\n 'value': ${outputs[1]},\\n 'dataType': 'torch.tensor',\\n 'metadata': {\\n 'colorChannel': 'rgb',\\n 'channelOrder': 'channelFirst',\\n 'isMiniBatched': False,\\n 'intensityRange': '0-255',\\n 'device': 'cpu'\\n }\\n}\\n${outputs[0]}`;\n if (inputs[2] === 'ImageReadMode.GRAY')\n return `${outputs[1]} = {\\n 'value': ${outputs[1]},\\n 'dataType': 'torch.tensor',\\n 'metadata': {\\n 'colorChannel': 'grayscale',\\n 'channelOrder': 'channelFirst',\\n 'isMiniBatched': False,\\n 'intensityRange': '0-255',\\n 'device': 'cpu'\\n }\\n}\\n${outputs[0]}`;\n}",
"inputs": {
"execIn": {
"title": "execIn",
"tooltip": "execIn",
"dataType": "exec",
"showWidget": false,
"showTitle": false
},
"path": {
"title": "path",
"dataType": "string",
"tooltip": "path(str) - path of the JPEG or PNG image."
},
"mode": {
"title": "mode",
"dataType": "imageio.ImageReadMode",
"defaultValue": "ImageReadMode.RGB",
"tooltip": "mode(ImageReadMode) - The read mode used for optionally converting the image. Default: ImageReadMode.UNCHANGED."
}
},
"outputs": {
"execOut": {
"title": "execOut",
"tooltip": "execOut",
"dataType": "exec",
"showWidget": false,
"showTitle": false
},
"image": {
"title": "image",
"dataType": "image",
"defaultValue": {
"dataType": "torch.tensor"
},
"tooltip": "{dataType: torch.tensor, value, layout: [chw], colorMode: [rgb, grayscale], intensityRange: 0-255' device: cpu}"
}
}
}
}
}
we first determine the code snippet that we want the node to provide :
then we add ${}
syntax on it :
code generator:
function code(inputs, outputs, node, generator) {
return `${outputs[1]} = ij.py.initialize_numpy_image(ij.io().open(${inputs[1]}))\n${outputs[1]} = ij.py.rai_to_numpy(ij.io().open(${inputs[1]}),${outputs[1]})\nij.py.show(${outputs[1]})\n${outputs[1]} = {\n 'value': ${outputs[1]},\n 'dataType': 'torch.tensor',\n 'metadata': {\n 'colorChannel': 'rgb',\n 'channelOrder': 'channelFirst',\n 'isMiniBatched': False,\n 'intensityRange': '0-255',\n 'device': 'cpu'\n }\n}\n${outputs[0]}`;
}
Intermediate Image Display on Node
:tif
format which supported by ImageJ, not by original pytorch librarynumpy.ndarray
, a image type that should be supported in VP
numpy
and the numpy
we get from PyImageJ
metadata
for numpy
wrong: we wrote :
n_1_image = {
'value': n_1_image,
'dataType': 'torch.tensor',
'metadata': {
'colorChannel': 'rgb',
'channelOrder': 'channelFirst',
'isMiniBatched': False,
'intensityRange': '0-255',
'device': 'cpu'
}
}
n_1_image = {
'value': n_1_image,
'dataType': '**numpy.ndarray**',
'metadata': {
'colorChannel': 'rgb',
'channelOrder': '**channelLast**',
'isMiniBatched': False,
'intensityRange': '0-255',
'device': 'cpu'
}
}
CaptureImageCode
structure:CaptureImageCode
structure:1. The conversion part:
The start conversion type is numpy.ndarray
, the destination type is current
we should know the RGB
numpy.ndarray
has three dimension such as the following jpg
image (1500,1063,3)
however the single-channel
tif image we read by ij.io().open()
has shape as (250,250), we don't have the third dimension
2. The PIL image part:
In CaptureImageCode
function returns a code snippet that contains VP import
, captureImageFunction
, commInJupyter
, convertFunctions
we get a pillow image(PIL) from function Image.fromarray()
This functions also requires numpy
array input
numpy
shape:colormapping
in matlibplotcolorchannel
is missing, because there's no color channel for grayscale imagecolor
on a grayscale picture, but actually the color comes from colormapping
in matlibplot
https://matplotlib.org/stable/gallery/color/colormap_reference.html
The picture without color mapping should look like:
CaptureImageCode
inside problems caused by non- unique dom id
are in Issue https://github.com/Max-ChenFei/VPE_IP/issues/70
is it caused by
[x] wrong PIL conversion?
[ ] wrong KG conversion?
[ ] wrong encode and decode?
jpg
image read by ori read_image_io
's case's PIL image
jpg
image to PIL
image should not change how it looksdtype
or shape
or range
could be differentunit8
dtype with range 0,255
2.1 Our Java object tif image we get from PyImage, then converted to numpy array has different type and range:
tif
image could not be correctly displayed after Image.fromArray()
array ->PIL conversion- Solution
we normalized the array into range [0,255]
and dtype uint8
by:
np_img_1707918575145['value'] = ((np_img_1707918575145['value'] - np_img_1707918575145['value'].min()) / (np_img_1707918575145['value'].max() - np_img_1707918575145['value'].min()) * 255).astype('uint8')
then the image is correctly displayed after PIL conversion:
CaptureImaegCode
functions:we should only keep the desired node panel as Watched
and unwatch all the rest node in other cells, notebooks, in the whole jupyter environment to keep its unique dom_id
watch
nodewe should normalize the image array into range [0,225]
and type uint8
with code like
np_img_1707918575145['value'] = ((np_img_1707918575145['value'] - np_img_1707918575145['value'].min()) / (np_img_1707918575145['value'].max() - np_img_1707918575145['value'].min()) * 255).astype('uint8')
so it could be correctly displayed
normalization
part in every node?CaptureImageFunction
before its PIL conversion Image.fromArray
CaptureImageFunction
itself so I'll just normalize the output numpy array of our IJ_VP_Node
Op
filterscode
and the display
workIJ_VP_Node
as a start/template, and add one Op to process the image
IJ_Mean_Filter_Node
that has input image
, radius
gray
cmpaIJ_otsu_segmentation
node-I think for segmention, this image is a better example:
https://imagej.net/plugins/auto-threshold#available-methods
Node
with different option
in node panelIJ_Read
Node before moving on to Op
We want the current IJ_read_image
node to
show grayscale
image without matplotlib's default color mapping
cmap=gray
for grayscale
image with 2D shape(x,y)show color tif
image and RGB jpg
image as it is(with shape x,y,channel)
IJ_read_image
, when reading jpg
image, we get numpy array with shape (channel, x, y), channel is in the first order of dimensionnumpy
array are ChannelLast
show helping messages for time-series with shape (x, y, channel, time)
show introductory messages before every image display in jupyter:
Original Image:
, After Mean Filter
, After Segmentation
def dump_info(image):
"""A handy function to print details of an image object."""
name = image.name if hasattr(image, 'name') else None # xarray
if name is None and hasattr(image, 'getName'): name = image.getName() # Dataset
if name is None and hasattr(image, 'getTitle'): name = image.getTitle() # ImagePlus
print(f" name: {name or 'N/A'}")
print(f" type: {type(image)}")
print(f"dtype: {image.dtype if hasattr(image, 'dtype') else 'N/A'}")
print(f"shape: {image.shape}")
print(f" dims: {image.dims if hasattr(image, 'dims') else 'N/A'}")
dump_info(dataset)
**
![image](https://github.com/Max-ChenFei/VPE_IP/assets/8528052/cfd18293-5a8d-413f-aa30-3c47899eb4fc)
**
---
**We would like our `IJ_read_image` to work as follows**
![image](https://github.com/Max-ChenFei/VPE_IP/assets/8528052/e18fb088-3b82-4b7c-9841-2a7d2f549915)
IJ_read_image
:import imagej
ij = imagej.init('sc.fiji:fiji:2.14.0')
import numpy as np
tif_image = ij.py.initialize_numpy_image(ij.io().open('retina.jpg'))
n_1_image = ij.py.rai_to_numpy(ij.io().open('retina.jpg'),tif_image)
number_of_dims = np.ndim(n_1_image)
def image_info(image):
# A handy function to print details of an image object.
print("--- Image Information ---")
name = image.name if hasattr(image, 'name') else None # xarray
if name is None and hasattr(image, 'getName'): name = image.getName() # Dataset
if name is None and hasattr(image, 'getTitle'): name = image.getTitle() # ImagePlus
print(f" type: {type(image)}")
print(f"dtype: {image.dtype if hasattr(image, 'dtype') else 'N/A'}")
print(f"shape: {image.shape}")
print(f" dims: {image.dims if hasattr(image, 'dims') else 'N/A'}")
print ("-------------------------")
image_info(n_1_image);
if number_of_dims == 2:
#Grayscale image
print("Grayscale image:")
ij.py.show(n_1_image,cmap='gray')
if number_of_dims == 3:
#RGB image
print("Input Image:")
#transpose (channel,x,y) -> (x,y,channel)
n_1_image = np.transpose(n_1_image, (1, 2, 0))
print(n_1_image.shape)
ij.py.show(n_1_image)
if number_of_dims == 4:
#time series image
print("Currently we focus on 2d images, please slice the 3d image for displaying in jupyter")
#Normalization to range (0,255), uint8
n_1_image = ((n_1_image - n_1_image.min()) / (n_1_image.max() - n_1_image.min())* 255).astype('uint8')
color_channel = 'grayscale' if number_of_dims == 2 else 'rgb'
n_1_image = {
'value': n_1_image,
'dataType': 'numpy.ndarray',
'metadata': {
'colorChannel': color_channel,
'channelOrder': 'channelLast',
'isMiniBatched': False,
'intensityRange': '0-255',
'device': 'cpu'
}
}
Display Color tif:
Display Grayscale tif without color mapping:
IJ_Read_Image
Code --> Nodeenum
option in node panel for colorChannel
\n
to each line of code and replace the input and output manually format_code() in json_node_generator
def format_code(code, input1, output1):
# Replace the input and output names in the code
code = code.replace("'{}'".format(input1), "${inputs[1]}")
code = code.replace(output1, "${outputs[1]}")
# Add a visible '\n' at the end of each line
code = '\\n'.join(code.strip().splitlines()) + '\\n'
code += '${outputs[0]}'
return code
# Test the function
code = """
tif_image = ij.py.initialize_numpy_image(ij.io().open('retina.jpg'))
n_1_image = ij.py.rai_to_numpy(ij.io().open('retina.jpg'),tif_image)
n_1_image = ((n_1_image - n_1_image.min()) / (n_1_image.max() - n_1_image.min())* 255).astype('uint8')
ij.py.show(n_1_image)
n_1_image = {
'value': n_1_image,
'dataType': 'numpy.ndarray',
'metadata': {
'colorChannel': 'grayscale',
'channelOrder': 'channelLast',
'isMiniBatched': False,
'intensityRange': '0-255',
'device': 'cpu'
}
}
"""
formatted_code = format_code(code, "retina.jpg", "n_1_image")
print(formatted_code)
tif_image = ij.py.initialize_numpy_image(ij.io().open(${inputs[1]}))\n${outputs[1]} = ij.py.rai_to_numpy(ij.io().open(${inputs[1]}),tif_image)\n${outputs[1]} = ((${outputs[1]} - ${outputs[1]}.min()) / (${outputs[1]}.max() - ${outputs[1]}.min())* 255).astype('uint8')\nij.py.show(${outputs[1]})\n${outputs[1]} = {\n 'value': ${outputs[1]},\n 'dataType': 'numpy.ndarray',\n 'metadata': {\n 'colorChannel': 'grayscale',\n 'channelOrder': 'channelLast',\n 'isMiniBatched': False,\n 'intensityRange': '0-255',\n 'device': 'cpu'\n }\n}\n${outputs[0]}
VP_IJ_node
, if so, we'll use this to generate the return of codeGenerator
IJ_Op_VP_node
template and auto-generation only for this partformat_code()
only generator part of JSONOp
plugin, IJ1 Marco
plugin , is it possible that we make an auto_Op_Node_generator
etc. based on a template?- Enhancement of Auto_Json_generation is in Issue https://github.com/Max-ChenFei/VPE_IP/issues/72
Copied textual Code
import imagej
ij = imagej.init('sc.fiji:fiji:2.14.0')
import numpy as np
tif_image = ij.py.initialize_numpy_image(ij.io().open('retina.jpg'))
n_1_image = ij.py.rai_to_numpy(ij.io().open('retina.jpg'),tif_image)
number_of_dims = np.ndim(n_1_image)
def image_info(image):
# A handy function to print details of an image object.
print("--- Image Information ---")
name = image.name if hasattr(image, 'name') else None # xarray
if name is None and hasattr(image, 'getName'): name = image.getName() # Dataset
if name is None and hasattr(image, 'getTitle'): name = image.getTitle() # ImagePlus
print(f" type: {type(image)}")
print(f"dtype: {image.dtype if hasattr(image, 'dtype') else 'N/A'}")
print(f"shape: {image.shape}")
print(f" dims: {image.dims if hasattr(image, 'dims') else 'N/A'}")
print ("-------------------------")
image_info(n_1_image);
if number_of_dims == 2:
#Grayscale image
print("Grayscale image:")
ij.py.show(n_1_image,cmap='gray')
if number_of_dims == 3:
#RGB image
print("Input Image:")
#transpose (channel,x,y) -> (x,y,channel)
n_1_image = np.transpose(n_1_image, (1, 2, 0))
print(n_1_image.shape)
ij.py.show(n_1_image)
if number_of_dims == 4:
#time series image
print("Currently we focus on 2d images, please slice the 3d image for displaying in jupyter")
#Normalization to range (0,255), uint8
n_1_image = ((n_1_image - n_1_image.min()) / (n_1_image.max() - n_1_image.min())* 255).astype('uint8')
color_channel = 'grayscale' if number_of_dims == 2 else 'rgb'
n_1_image = {
'value': n_1_image,
'dataType': 'numpy.ndarray',
'metadata': {
'colorChannel': color_channel,
'channelOrder': 'channelLast',
'isMiniBatched': False,
'intensityRange': '0-255',
'device': 'cpu'
}
}
Partial_Json_generator:
def format_code(code, input1, output1):
# Replace the input and output names in the code
code = code.replace("'{}'".format(input1), "${inputs[1]}")
code = code.replace(output1, "${outputs[1]}")
# Add a visible '\n' at the end of each line
code = '\\n'.join(code.strip().splitlines()) + '\\n'
code += '${outputs[0]}'
# Add backticks at the beginning and end of the code
code = '`' + code + '`'
return code
# Test real code
code = """
tif_image = ij.py.initialize_numpy_image(ij.io().open('test_grayscale_image.tif'))
n_1_image = ij.py.rai_to_numpy(ij.io().open('test_grayscale_image.tif'),tif_image)
number_of_dims = np.ndim(n_1_image)
def image_info(image):
# A handy function to print details of an image object.
print("--- Image Information ---")
name = image.name if hasattr(image, 'name') else None # xarray
if name is None and hasattr(image, 'getName'): name = image.getName() # Dataset
if name is None and hasattr(image, 'getTitle'): name = image.getTitle() # ImagePlus
print(f" type: {type(image)}")
print(f"dtype: {image.dtype if hasattr(image, 'dtype') else 'N/A'}")
print(f"shape: {image.shape}")
print(f" dims: {image.dims if hasattr(image, 'dims') else 'N/A'}")
print ("-------------------------")
image_info(n_1_image);
if number_of_dims == 2:
#Grayscale image
print("Grayscale image:")
ij.py.show(n_1_image,cmap='gray')
if number_of_dims == 3:
#RGB image
print("Input Image:")
#transpose (channel,x,y) -> (x,y,channel)
n_1_image = np.transpose(n_1_image, (1, 2, 0))
print(n_1_image.shape)
ij.py.show(n_1_image)
if number_of_dims == 4:
#time series image
print("Currently we focus on 2d images, please slice the 3d image for displaying in jupyter")
#Normalization to range (0,255), uint8
n_1_image = ((n_1_image - n_1_image.min()) / (n_1_image.max() - n_1_image.min())* 255).astype('uint8')
color_channel = 'grayscale' if number_of_dims == 2 else 'rgb'
n_1_image = {
'value': n_1_image,
'dataType': 'numpy.ndarray',
'metadata': {
'colorChannel': color_channel,
'channelOrder': 'channelLast',
'isMiniBatched': False,
'intensityRange': '0-255',
'device': 'cpu'
}
}
"""
formatted_code = format_code(code, "test_grayscale_image.tif", "n_1_image")
print(formatted_code)
json_generator_output:
`tif_image = ij.py.initialize_numpy_image(ij.io().open(${inputs[1]}))\n${outputs[1]} = ij.py.rai_to_numpy(ij.io().open(${inputs[1]}),tif_image)\nnumber_of_dims = np.ndim(${outputs[1]})\ndef image_info(image):\n # A handy function to print details of an image object.\n print("--- Image Information ---")\n name = image.name if hasattr(image, 'name') else None # xarray\n if name is None and hasattr(image, 'getName'): name = image.getName() # Dataset\n if name is None and hasattr(image, 'getTitle'): name = image.getTitle() # ImagePlus\n print(f" type: {type(image)}")\n print(f"dtype: {image.dtype if hasattr(image, 'dtype') else 'N/A'}")\n print(f"shape: {image.shape}")\n print(f" dims: {image.dims if hasattr(image, 'dims') else 'N/A'}")\n print ("-------------------------")\nimage_info(${outputs[1]});\nif number_of_dims == 2:\n #Grayscale image\n print("Grayscale image:")\n ij.py.show(${outputs[1]},cmap='gray')\nif number_of_dims == 3:\n #RGB image \n print("Input Image:")\n #transpose (channel,x,y) -> (x,y,channel)\n ${outputs[1]} = np.transpose(${outputs[1]}, (1, 2, 0))\n print(${outputs[1]}.shape)\n ij.py.show(${outputs[1]})\nif number_of_dims == 4:\n #time series image\n print("Currently we focus on 2d images, please slice the 3d image for displaying in jupyter")\n#Normalization to range (0,255), uint8 \n${outputs[1]} = ((${outputs[1]} - ${outputs[1]}.min()) / (${outputs[1]}.max() - ${outputs[1]}.min())* 255).astype('uint8')\ncolor_channel = 'grayscale' if number_of_dims == 2 else 'rgb'\n${outputs[1]} = {\n 'value': ${outputs[1]},\n 'dataType': 'numpy.ndarray',\n 'metadata': {\n 'colorChannel': color_channel,\n 'channelOrder': 'channelLast',\n 'isMiniBatched': False,\n 'intensityRange': '0-255',\n 'device': 'cpu'\n }\n}\n${outputs[0]}`
node name
and node type
in json object should be the same, title
doesn't matter for Node but is important for user to read
"""
1.2 Particles Analysis: (https://py.imagej.net/en/latest/Puncta-Segmentation.html)
1.4 Frangi vesselness
from a MRA brain image:
(https://github.com/imagej/tutorials/blob/master/notebooks/1-Using-ImageJ/2-ImageJ-Ops.ipynb)
1.5 Thresholding
, tons of thresholding op:
https://github.com/imagej/tutorials/blob/master/notebooks/1-Using-ImageJ/2-ImageJ-Ops.ipynb
we should use test image here: https://imagej.net/plugins/auto-threshold#available-methods
1.6 Morphology
https://github.com/imagej/tutorials/blob/master/notebooks/1-Using-ImageJ/2-ImageJ-Ops.ipynb
(-)2. More advanced cellpose segmentation for future case: (https://py.imagej.net/en/latest/Cellpose-StarDist-Segmentation.html)
(x)3. More more advanced texture analysis and add red rectangle on image: https://py.imagej.net/en/latest/GLCM.html#using-pyimagej-and-imagej-ops-for-glcm-texture-analysis
3d_image_slicing
:
cell_segmentation_slicing6.json
edge_preserve_smoothing
cell_segmentation_slicing6_edge7.json
otsu_segmentation
cell_segmentation_s6_e7_otsu1.json
4.1 morphology filter_dilation
cell_segmentation_s6_e7_o1_morph3.json
4.2 morphology filter_fillHoles
cell_segmentation_s6_e7_o1_m3_fillHole1.json
watershed labeling
we might problems at watershed labeling:
In short, the ImgLabeling java object is not supported in PyImageJ and it's hard for us to convert it into normal python datatype
then we can not display it on NodePanel via PIL
but actually it can still shown in the cell:
but the result is also not as good as it should be
so we'll omit it first and fix it later
First VP ImageJ node example:
What code should be in the
VP import
part ?We have
Python normal import
: to import python librariesJava Jimport
: to import java librariesImageJ initialization
to initialize PyImageJ and ImageJ in the beginning, this part is necessary for all ImageJ functions:initialize ImageJ
ij = imagej.init('sc.fiji:fiji:2.14.0') print(f"ImageJ version: {ij.getVersion()}")
Current Solution
VP import
part for ImageJ should be :Python normal import
ImageJ initialization
Java JImport
should be in theVP code
partReasons
VP import
is stored asSet
to avoid duplicationVP code
is stored asstring
which can not avoid duplicationImageJ Initialization
is required at the beginning of the code before running any ImageJ functions, so for everyfunction ImageJ node
, we'll have theImageJ Initialization
part.We don't want to connect to ImageJ gate multiple times, so
ImageJ Initialization
should be not be duplicated.Thus we make the decision of putting the
ImageJ Initialization
part intoVP import
partSmall concerns
VP import
:ImageJ Initialization
part in theVP import
ij = imagej.init('sc.fiji:fiji:2.14.0')
would locatePython normal imports
, would it still work?