Running Viv in ImJoy and Jupyter notebooks

[oeway]

Introduced by @gzuidhof , I started to play with this viewer and it feels great! Thanks for the great work.

Now wondering if you will be interested in integrating some changes to support running the viewer as an ImJoy plugin and used in Jupyter notebooks:

I made a little demo in my forked repo: https://github.com/oeway/vitessce-image-viewer

And you can try it on Binder here:

The changes to the code is very minimal, this is the diff.

As you can see, for demonstration purpose, I only implemented a loadSource function. This can be extended further, for example, we can also send numpy array directly to the viewer (see another binder demo here). We can also support sending a getItem function from Python to JS and used as a zarr.js store (as consulted with @gzuidhof), like that one can open an zarr array in python and show it in the viewer, either in ImJoy or a jupyter notebook. For more details, see imjoy-rpc and imjoy-core.

What do you think?

[oeway]

UPDATE: Now you can use Viv to visualize your Zarr array or group opened in Python (in ImJoy or a Jupyter notebook).

It turns out pretty easy to implement, we just need an additional imjoy-rpc codec to encode the zarr array/group, then we can just send them directly to the loadSource function.

import zarr
from tifffile import imread
image = imread('cell_membranes.tif')
image = (image*65536).astype('uint16')
z_array = zarr.array(image, chunks=(50, 50))

staticInfo = {
  # here we pass directly the zarr array, because we have registered a codec
  "url": z_array,
  "isPublic": False,
  "initialViewState": {
    "zoom": -1,
    "target": [100, 100]
  },
  "dimensions": [
    {
      "field": 'mz',
      "type": 'ordinal',
      "values": [str(i) for i in range(60)]
    },
    { "field": 'y', "type": 'quantitative', "values": None },
    { "field": 'x', "type": 'quantitative', "values": None }
  ],
  'selections': [{'mz': 21}],
    'colormap': 'viridis',
  "description": 'Membrane (zarr)'
}

from imjoy import api
import asyncio

class ImJoyPlugin():
    async def setup(self):
        pass

    async def run(self, ctx):
        viewer = await api.createWindow(type="vitessce-image-viewer", src="https://oeway.github.io/vitessce-image-viewer/")
        #await asyncio.sleep(10)
        await viewer.loadSource('static', staticInfo)

api.export(ImJoyPlugin())

As you see, we simply pass the encoded zarr store interface as url, and I had to do some minor fix in the viewer to treat the url as a zarr.js store.

Here is a demo Jupyter notebook that implements this along with other example datasets including zarr, pyramid, omezarr etc.

I have updated the binder demo above, try it yourself here:

[ilan-gold]

Hi @oeway this is all very exciting, and I'm glad you have really pushed forward with this. I know @manzt has been working on Jupyter integration recently so he might be able to speak to the nuts and bolts of all this. I can try to speak at a high-level about this, though.

It looks like you have hacked something via a fork of our repo, but we can work to make something a little more deployable for you, like an npm package that you could then wrap more transparently in a imjoy-viv repo or the like.

To that end, Vitessce at vitessce.io, for which this was created, has the ability to overlay cell segmentations via polygon boundaries, which I imagine you may be interested in. Its image viewer is/will likely be more mature than just our demo here over time - we hope to integrate channel distributions via histograms, among other things. It also is already a standalone package (exposing React components including for the image viewer) that you can import directly, and wrap in your imjoy api. If you don't want to use it, though, we can explore making this repo's demo a little cleaner/more deployable for your use.

More forward looking, we have 3D viewing in the works here (on its own branch for now), but I think you already have something for this purpose. If you're interested though, let us know. Also, I have fiddled around with on-the-fly cell-segmantaion via tensorflowjs (that then displays as an overlaid layer). This would require some tinkering to officially support, but it seems to work at a high level.

This is all take-it-or-leave-it stuff - it's just great to have someone as excited about this as we are! Let us know what interests you!

[oeway]

Hey @ilan-gold, sounds great! It seems we have very similar goal towards visualising big dataset, 3D, jupyter and deep learning in the browser. I think we should interact more and joint the effort, you have done great job in the viewer, especially the support of Zarr/TIFF/OME-Zarr, Also the Vitessce tool seems super cool. I think it would be nice if you could consider integrate imjoy-rpc to Vitessce. As you see above, with minimal changes and it gives the ability to feed data from imjoy or jupyter notebook, it give the user a lot more flexibility.

It looks like you have hacked something via a fork of our repo, but we can work to make something a little more deployable for you, like an npm package that you could then wrap more transparently in a imjoy-viv repo or the like.

Yes, that would great. For the jupyter notebook support, I had some small fix to allow passing a zarr.js store object instead of an URL, would be nice to expose that in a standard way.

We started to work on the Kaibu viewer, and we would like to integrate Viv into it as a layer, just like what we did with itk-vtk-viewer. I tagged you in this PR, and I had to make a few other hacks to expose deck object and viewState (see this branch). I am new to React (mostly use Vuejs in my project), the changes I made is likely very hacky, please just consider it and implement the way you like.

Also I found deck.gl seems super powerful, was not aware that before. However, it seems it doesn't run on iOS, and there is an ongoing PR for safari support, which is great!

How about having a meeting together to discuss these? @ilan-gold @manzt

[ilan-gold]

Yes, I think a meeting would be good. @manzt and I are both free all day tomorrow except for about 15 minutes at 9am EST.

[manzt]

This is really exciting to see. It would be great to discuss further. Looking forward to it!

[oeway]

Great, let's do it tomorrow Tuesday 16th, 9:00 AM Boston time (3pm at Stockholm) then, here is the zoom link: https://kth-se.zoom.us/j/64375732906

Others who are interested in this topic please also feel free to join the call.

Looking forward!

[manzt]

Great to meet today, @oeway ! Thanks again for walking us through ImJoy + Kaibu. Would you be able to share the binder notebook using the Kaibu ImJoy Plugin? I'd like to look at how callbacks are defined on the python side. I have much less experience with async in python. Second, have you tried imjoy in a google colab before?

[oeway]

Nice to talk to you guys today.

Would you be able to share the binder notebook using the Kaibu ImJoy Plugin? I'd like to look at how callbacks are defined on the python side. I have much less experience with async in python.

Jupyter notebook example, notebook :

Please also note that there is a current limitation with jupyter notebook that the execution will be blocked if you try to run directly await in a cell, see this issue. Instead of doing await my_async_function() directly in a cell, you can do asyncio.ensure_future(my_async_function()).

Second, have you tried imjoy in a google colab before?

Yes, we have been trying google colab, and the difficult part is that they don't support standard jupyter comm message (which we use for imjoy). However, I have come up with a solution with their api, didn't have time to come back to that, but it should be possible to directly add support in imjoy-rpc.

---

Here are some further summary information:

For integrating ImJoy-rpc to vitessce, here are a few places to get started:

• details about the imjoy-rpc library: https://github.com/imjoy-team/imjoy-rpc
• the jupyter notebooke extension: https://github.com/imjoy-team/imjoy-jupyter-extension
• some instructions about integrating imjoy to your project is here: https://github.com/imjoy-team/imjoy-core
• the example in my fork here

So I think would nice to have imjoy-rpc integrated to your standalone version of the vitessce-image-viewer or vitessce.io for also other data types in single cell analsysis.

For Kaibu:

• Jupyter notebook example, notebook
• The PR for adding Vivi is here
• The branch in my fork of VIv that contains changes for exposing deck and view state etc. is here

[manzt]

Also have a look at vizarr! https://github.com/hms-dbmi/vizarr