Kitware / itk-vtk-viewer

2D / 3D web image, mesh, and point set viewer using itk-wasm and vtk.js
https://kitware.github.io/itk-vtk-viewer/
BSD 3-Clause "New" or "Revised" License
210 stars 64 forks source link

Fix numpy array support for the ImJoy API #344

Closed oeway closed 3 years ago

oeway commented 4 years ago

It appears that the numpy array isn't working for the ImJoy API:

To reproduce this issue, run pip install imjoy-jupyter-extension then start a jupyter notebook: jupyter notebook, make sure you see an ImJoy icon in the toolbar of your notebook. Now, copy and paste the following code and run:

import imageio
import numpy as np
from imjoy_rpc import api

class ImJoyPlugin():
    def setup(self):
        api.log('plugin initialized')

    async def run(self, ctx):
        viewer = await api.showDialog(src="https://kitware.github.io/itk-vtk-viewer/app/")
        # show a 3D volume
        image_array = np.random.randint(0, 255, [10,10,10], dtype='uint8')

        # show a 2D image
        # image_array = imageio.imread('imageio:chelsea.png')

        # `setImage` is an api function exposed by itk-vtk-viewer
        # see here: https://kitware.github.io/itk-vtk-viewer/docs/imjoy.html#API-functions
        await viewer.setImage(image_array)

api.export(ImJoyPlugin())

If you now open the browser console, there is an error rpc.js:532 TypeError: Cannot read property 'dimension' of undefined.

This happens because the conversion between ndarray to itkimage is not functioning (code here). The current implementation treat the input as a scijs/ndarray, but imjoy-rpc won't convert numpy array to scijs/ndarray and itk-vtk-viewer will get the encoded ndarray({_rtype: 'ndarray', _rshape: ..., _rvalue: ..., _rdtype: ...}). ( It can do numjs and tensorflow.js Tensor, see here.

In order to make it work with numpy array, we will need to convert the imjoy-rpc encoded ndarray into itk-image. Therefore I suggest we change the ndarrayToItkImage function to something like I have before in my forked version. And we activate this conversion by checking if image._rtype === 'ndarray'.

thewtex commented 3 years ago

:tada: This issue has been resolved in version 10.9.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: