search

Kitware / vtk-js

Visualization Toolkit for the Web
https://kitware.github.io/vtk-js/
BSD 3-Clause "New" or "Revised" License
1.23k stars 370 forks source link

Use ImageReslice for ImageMapper XYZ slicing modes when dataset isn't axis-aligned #1720

Open agirault opened 3 years ago

agirault commented 3 years ago

The current behavior of the vtkImageMapper is:

That was when initially working on vtk.js when we did not yet have vtkImageReslice. Now that we do (for a while actually), it seems that XYZ slicing should reslice if the dataset is non-axis aligned, instead of this.

Any thoughts @finetjul @laurennlam @luciemac @floryst ?

finetjul commented 3 years ago

vtkImageReslice is CPU.

Isn't ImageMapper GPU ?

I'm not sure what prevents the non-axis aligned XYZ ?

Fyi: https://react-vtkjs-viewport.netlify.app/rotatable-crosshairs

floryst commented 3 years ago

ImageMapper isn't GPU. The slicing happens here, in vtkOpenGLImageMapper.

floryst commented 3 years ago

Re: react-vtkjs-viewport rotatable-crosshairs. They are using InteractorMPRSlice for that, which is just volume rendering with the camera clipping range used to define a thin slab.

agirault commented 3 years ago

react-vtkjs-viewport rotatable-crosshairs. They are using InteractorMPRSlice for that, which is just volume rendering with the camera clipping range used to define a thin slab.

That's beautiful... This is the way! What's the fallback of that method? Can we do picking on it?

I'm not sure what prevents the non-axis aligned XYZ ?

From my understanding, the vtkImageMapper can only read along the image axis (no interpolation), and when we do XYZ, we basically find the closest image axis. My thought was that we could either:

  1. modify the vtkImageMapper to reslice if trying to get data that's not axis-aligned, or
  2. remove XYZ slicing entirely from the vtkImageMapper, and the application is responsible for reslicing it before feeding it to the mapper.

Does one of you know what VTK C++ does? cc: @sankhesh @martinken

floryst commented 3 years ago

We should be able to do picking on it, which is why I think we can do better with the clipping planes on the mapper. That way we can have other objects in the scene that may exist beyond the slice bounds.

sankhesh commented 3 years ago

They are using InteractorMPRSlice for that, which is just volume rendering with the camera clipping range used to define a thin slab.

I wouldn't do that. Limiting the camera clip range would not only just limit the camera view to the image, it wouldn't be accurate.

The C++ vtkImageMapper class doesn't do oblique slicing. I don't think either of vtkImageMapper and vtkImageReslice support the oriented image data.

agirault commented 3 years ago

Limiting the camera clip range would not only just limit the camera view to the image, it wouldn't be accurate.

@sankhesh we should discuss that in more detail to see why/if that's the case: the OHIF folks are doing this with vtk.js using a VolumeMapper, and it seems to look and work great.

cc: @lassoan @pieper

The C++ vtkImageMapper class doesn't do oblique slicing. I don't think either of vtkImageMapper and vtkImageReslice support the oriented image data.

If/when you plan to add support for it in C++, what architecture would you be thinking of? Being able to set an arbitrary axis on the ImageMapper and compute values along that orthogonal place? Using ImageReslice inside the ImageMapper? Before? I actually don't know what Slicer does on top of my head.

pieper commented 3 years ago

Yes, for OHIF we like the volume + clip plane approach in the GPU. It naturally extends to slab rendering and is super fast. Also we are able to cleanly do screen space operations in the shader like controlling the thickness of a segmentation outline. You can try here by clicking on the "2D MPR" button.

Slicer uses vtkImageReslice to put all layers into screen space and then we do the compositing there.

sankhesh commented 3 years ago

That sounds like using a couple of clip planes with the volume mapper rather than playing with the camera clip range. That is indeed a good approach to do fast slicing on the GPU.

agirault commented 3 years ago

Thank you @sankhesh @pieper !

Back to the main topic: XYZ slicing in vtkImageMapper right now is just inaccurate for non-axis aligned datasets. Options I can think of are:

  1. Drop it from vtkImageMapper: users can use vtkImageReslice (CPU) beforehand or the Volume Mapper (GPU) with clip planes instead
  2. Update the vtkImageMapper to do the reslicing internally if needed instead of closestAxis
  3. Do nothing, but update the warning for non-axis aligned datasets rendered by vtkImageMapper to direct developers to the other solutions mentioned in (1)