Streaming?

[skitaeff]

I wonder is the memory model may allow streaming? We've developed a FITS to JPX encoder (supports WCS) and a server backend to stream JPEG2000 imagery via JPIP. This enables easy remote visualisation of TB cubes at different fidelities and resolutions progressively streaming the data from a server. We are looking for suitable client program that would a) support needed functionality for HI cubes, and b) has the memory model and suitable API that would be able to support streaming as well as usual static files. Thanks!

[Punzo]

Hi Slava, the client-server feature is not available in 3DSlicer. We had some plans to add it (together to have also catalogs client-server features, i.e. VO and SAMP protocol), however the SlicerAstro financial support will end this August (i.e., I'll have to find a new job :) ). Therefore, all the client-server options will not be implemented (point 6 in the roadmap and readme in the wiki).

So making it short, the answer is no. I agree that it will be a nice addition, if you have the resources (manpower or financial) I'll be glad to help you in adding such feature.

I don't know the needs of your user case, but you may have also a look to Paraview (despite can't read fits file and no WCS, etc.), because of Paraview has the skill to connect to a server and run the VTK rendering on the server and getting back the image on the client GUI (https://www.paraview.org/Wiki/Setting_up_a_ParaView_Server). However, I don't know the details of the implementation and if you can factorize and customize the classes to use your code and protocols.

[skitaeff]

Hi Davide,

I'm sorry to hear about the lack of funding to continue the work. Apparently it is quite a common problem in astronomy.

Our use case is SKA HI cubes. I'm sure you are aware that they are big.

I did have a look into Paraview. Server rendering might indeed be an interesting feature but it has some limitations. If you want to inspect a 1-3TB cube quickly, 3D rendering might be secondary issue to the bandwidth and I/O issue. JPEG2000 and JPIP don't have powerful 3D capability but they allow the bandwidth optimisation really well through the mechanisms of multiple quality layers, adaptive encoding (ROI or precincts) and progressive transfer. We've build a prototype called SKUAREVIEW that allows browsing TB cubes with an ease via even wifi.

Streaming requires a client to implement a different memory model. It has to have a constantly updated cash to support streaming, and a signalling mechanism to update the screen. I could not find if Paraview can support such a model, and thought maybe you looked into this for the SlicerAstro. I can see that you've enabled some useful functionality already that maybe if the Slicer can support a dynamic update of memory to implement JPIP we may contribute by developing the JPIP module.

[Punzo]

I'm sorry to hear about the lack of funding to continue the work. Apparently it is quite a common problem in astronomy.

It happens (:, but I am sorry that I'll not have too much time to dedicate to your use case, which it is interesting (see below).

Our use case is SKA HI cubes. I'm sure you are aware that they are big.

yeap (:

I did have a look into Paraview. Server rendering might indeed be an interesting feature but it has some limitations. If you want to inspect a 1-3TB cube quickly, 3D rendering might be secondary issue to the bandwidth and I/O issue. JPEG2000 and JPIP don't have powerful 3D capability but they allow the bandwidth optimisation really well through the mechanisms of multiple quality layers, adaptive encoding (ROI or precincts) and progressive transfer. We've build a prototype called SKUAREVIEW that allows browsing TB cubes with an ease via even wifi.

I agree with you. Moreover looking at the data in 2D for large datacubes, it may have a much higher ratio efficiency (looking for artifacts, etc.)/cost (server dimensions, for 3D visu of 1TB you need a cluster with a lot of GPUs (:., e.g. NVIDIA IndeX volume render on cluster of GPUs + Paraview as client https://developer.nvidia.com/index ). For reference I add SKUAREVIEW link http://skuareview.net/

Streaming requires a client to implement a different memory model. It has to have a constantly updated cash to support streaming, and a signalling mechanism to update the screen. I could not find if Paraview can support such a model, and thought maybe you looked into this for the SlicerAstro. I can see that you've enabled some useful functionality already that maybe if the Slicer can support a dynamic update of memory to implement JPIP we may contribute by developing the JPIP module.

The data in SlicerAstro are saved in the class vtkImageData: https://www.vtk.org/doc/nightly/html/classvtkImageData.html

and the object vtkImageData are allocated in the vtkMRMLAstroVolumeNode: https://codedocs.xyz/Punzo/SlicerAstro/classvtkMRMLAstroVolumeNode.html

For example, SlicerAstro allocates them at the moment when reading a FITS file here (which is a simple parsing of the data pointer): https://github.com/Punzo/SlicerAstro/blob/master/vtkFits/vtkFITSReader.cxx#L2892-L2989

However, VTK has a streaming class (not used by 3DSlicer and SlicerAstro at the moment): https://www.vtk.org/doc/nightly/html/classvtkImageDataStreamer.html

and here a python example how to use that https://blog.kitware.com/streaming-in-vtk-time/

I am not sure that the vtkImageDataStreaming class can help you (I have no knowledge how your code works. On the client part does it give a JPEG and the you visualize the JPEG? or a pointer to data?) and unfortunately I don't have the time to experiment by myself (I have pretty tight roadmap up to August to finalize a stable version of SlicerAstro). But if you can connect your code with this class (i.e., get the output of your code and essentially update the vtkImageData pointer to the data with it), I think we can easily do a module in SlicerAstro (I can help on this).

Please, let me know if you have further questions/feedback.

[skitaeff]

Thanks for the info! I'm going to look into this though I have no background in VTK whatsoever. Then perhaps a naive question. If both, Slicer and Paraview use VTK, so your code, are Astro modules portable to Paraview?

[Punzo]

Thanks for the info! I'm going to look into this though I have no background in VTK whatsoever. Then perhaps a naive question.

No problem! The answer is yes and no. The pure SlicerAstro VTK classes will need probably just few edits, e.g.: https://github.com/Punzo/SlicerAstro/tree/master/vtkFits https://github.com/Punzo/SlicerAstro/tree/master/vtkOpenGLFilters (but still there will be needed work to use them in Paraview infrastructure).

But all the others modules rely on the MRML node structure of 3DSlicer (objects that store the parameters of the data and of the GUI) https://www.slicer.org/wiki/Documentation/Nightly/Developers/MRML . A porting of them to paraview will not be easy at all.