Integration with OpenLayers

[oeway]

We have been using OpenLayers for displaying images in some of the ImJoy plugins (e.g.: https://imjoy.io/lite?plugin=oeway/ImJoy-Plugins:ImageAnnotator), and we have been benefiting from a well equipped 2D annotation features provided by the library.

Now I started to think whether we can combine the itk-vtk-viewer with OpenLayers, say, create a new CanvasLayer in OpenLayers, and use itk-vtk-viewer to manipulate that layer. Here is an example about d3 integration in OpenLayers: https://openlayers.org/en/latest/examples/d3.html Essentially, we need to provide a render function to get synchronized.

Could you please tell me what do you think about that? Or do you have a native way of supporting annotation in the viewer?

If that would work, I think it's gonna be a very powerful viewer with annotation support. Annotating 2D images should work without much tweaking, and for 3D data, we can at least support plane based annotation.

@thewtex

[thewtex]

We can certainly overlay a new canvas element, e.g. the scale bar that is new in https://imjoy.io/lite?plugin=oeway/ImJoy-Plugins:itk-vtk-viewer (near the bottom) is a separate canvas overlay.

That said, like you suggested are very powerful and necessary, we are developing native widgets for 3D annotation in vtk.js and itk.js, which need to be integrated. I will follow-up via email with details.

[oeway]

Hi @thewtex while waiting the native annotation widget, I started to hack a bit by combining itk-vtk-viewer with OpenLayers. The idea is to enable 2D annotation with OpenLayers when the viewer is in 2D mode or showing a plane. My current implementation used itk-vtk-viewer as custom layer in OpenLayers.

Here is what I got:

This is the live demo: https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/531c2f72ff712414f2229f9f8b522464/raw/6adfc0639aa3ca9c08aa64f74ba6daeec36f9f15/ImageViewer.imjoy.html The polygon annotation tool is enabled, so you can click on the screen to add new points to a polygon and close it by clicking the starting point.

As you will see, the viewer is not changing when you zoom or drag the plane, because it's not synchronized with OpenLayers yet (but the polygon annotations will change when you zoom). Would be great if you can help me with that. Specifically, I have the following two questions:

1. How to control the zooming and panning when showing a 2D plane? If you run the live demo and open your console, you can see the center coordinates and resolution provided by openlayers (generated here). I would like to use the center coordinates and resolution to control camera itk-vtk-viewer, such that the current view in openlayers matches itk-vtk-viewer.
2. Currently, the mouse events are handled by openlayers, which is good in 2D plane mode. However, in volumetric rendering mode, I would like to pass mouse events to the viewer. Could you please tell me if there is a mouse event handler in the viewer which I can manually pass the event to? I would like to forward the mouse event to the viewer in a function like this

Thanks in advance.

[oeway]

quick update: I manage to solve my second question, which is I can now forward the mouse events to the itk-vtk-viewer.

I added a button, so I can now enable and disable the annotations in OpenLayers:

https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/531c2f72ff712414f2229f9f8b522464/raw/fec2ebc921a30c9d388ea303f4072c630847d980/ImageViewer.imjoy.html

To move on, I would like to have some hint on how should we manipulate the camera to make the viewer match my openlayers coordinates.

[thewtex]

https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/531c2f72ff712414f2229f9f8b522464/raw/fec2ebc921a30c9d388ea303f4072c630847d980/ImageViewer.imjoy.html

Nice!

On my system (Linux / Chrome), the viewer slides in from the left:

This can be fixed by adding position to the div as discussed in https://github.com/Kitware/itk-vtk-viewer/issues/223

To move on, I would like to have some hint on how should we manipulate the camera to make the viewer match my openlayers coordinates.

Here is an example on how to get the camera parameters:

https://github.com/InsightSoftwareConsortium/itkwidgets/blob/dac67b0eeefe8b460b3fa79901166623e74dc294/js/lib/viewer.js#L717-L720

And more info on what they mean so they could be changed:

https://kitware.github.io/vtk-js/api/Rendering_Core_Camera.html

[oeway]

Hey, thanks for the tips, I will take a look and see if I can get it to work.

Now fixed the sliding bug: https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/531c2f72ff712414f2229f9f8b522464/raw/c782e71b9b941dff1d37f81b0507819776e13d2a/ImageViewer.imjoy.html

[oeway]

@thewtex BTW, could you also give me some hint on how to convert the pixel coordinates and pixel size on the screen to the coordinates in the render, and further to the settings of the camera parameters? What I would like to do is to get the coordinates from OpenLayers, convert to pixel coordinates and pixel size, and then somehow get camera settings of the itk-vtk-viewer.

[oeway]

Hey @thewtex I made it! A preliminary version which synchronizes the two coordinate system:

https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/531c2f72ff712414f2229f9f8b522464/raw/6598d390172ef3ad10300145b7e978e2e6753d85/ImageViewer.imjoy.html

However, while I can do camera. setParallelScale to set the scale of the viewer, I cannot find a way to shift/translate the camera, I tried:

const viewFocus = camera.getFocalPoint();
const viewPoint = camera.getPosition();
camera.setFocalPoint([viewFocus[0]-diff_x, viewFocus[1]-diff_y, viewFocus[2]]);
camera.setPosition([viewPoint[0]-diff_x, viewPoint[1]-diff_y, viewPoint[2]]);
viewProxy.getRenderer().getRenderWindow().render();

( code: https://gist.github.com/oeway/531c2f72ff712414f2229f9f8b522464/533a0226c86cc14f7995b9f0c72c71ce23a92a8e/#file-imageviewer-imjoy-html-L207-L208 ) The issue of the above code is that the entire display gone if call render(). Do you know why?

Another thing is for the Zplane, openlayers uses lower-left corner as it's origin, but the viewer uses top-left. I had to do:

viewer.setViewMode('ZPlane');
viewProxy.updateOrientation(2, 1, [0, 1, 0])

But I think I can fix this if I find a way to set the orientation in openlayers.

[thewtex]

More information on the camera parameters: https://kitware.github.io/vtk-js/api/Rendering_Core_Camera.html

The issue of the above code is that the entire display gone if call render(). Do you know why?

The camera clipping range may need to be set.

A few possible helpful snippets:

https://github.com/InsightSoftwareConsortium/itkwidgets/blob/dac67b0eeefe8b460b3fa79901166623e74dc294/js/lib/viewer.js#L1026-L1030

https://github.com/Kitware/itk-vtk-viewer/blob/0220d747dc8a9572d83e58fa01005110c8c58b6e/src/ItkVtkViewProxy.js#L233-L239

[oeway]

Hey @thewtex Thanks for your code snippets! I finally figured it out! It was because I passed an array instead of three arguments ;(

//wrong!
camera.setFocalPoint([xf, yf, zf])
camera.setPosition([xp, yp, zp]);

//correct
camera.setFocalPoint(xf, yf, zf)
camera.setPosition(xp, yp, zp);

Now it works like a charm : https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/531c2f72ff712414f2229f9f8b522464/raw/9d750604cefed030df83a86646fa537f1423915e/ImageViewer.imjoy.html

A huge thanks to you!