search

Kitware / resonanthpc

HPC-Enabled Pre- and Post-Processing with Jupyter
https://kitware.github.io/resonanthpc/
Apache License 2.0
6 stars 0 forks source link

Refactor/re-brand ipyparaview for use with VTK or ParaView backend #16

Open aashish24 opened 4 years ago

banesullivan commented 4 years ago

After talking with @thewtex, I thought it would be good to outline my thoughts/takeaways from our conversation on what would be a good direction for a 3D viz Jupyter-based framework and where I think there is potential for impact (for this project and the bigger scene).

I'd also like to ping (hopefully not spam) a few folks: @tjcorona and @jourdain

The TLDR; ipyparaview is a good proof of concept, but we should rebuild a more extensible frontend using existing work in vtk.js and itk-vtk-viewer for use inside/outside of Jupyter.

My goal here is to build a highly general and extensible frontend for 3D rendering in Jupyter (and other web) environments. This toolkit will enable pure server-side rendering and synchronized client-side rendering. It will interface to any server-side vtkRenderWindow (ParaView or VTK-Python) for streaming screenshots to the client or streaming data to a vtk.js renderer on the client.

Currently, it seems we should step back from ipyparaview. It's an excellent proof of concept, but much of the features like tracking camera movements and sending those back to VTK have been implemented more robustly in vtk.js (and ParaViewWeb??). Similarly, other relevant work has been done that is all currently a bit fragmented across a few projects. It looks to me as though the best route forward is to start from the ground up to build off of previous work and glue together what is currently in place. Starting from the ground up would also enable us to build out this tool in a way that it can be embedded in any web environment (not just Jupyter) so that we have a quick and easy way to deploy a 3D viz web application.

To move forward, I have the following list of next steps:

  1. use itk-vtk-viewer and wslink to prototype the synchronizable client side rendering
    • Specifically, see this
  2. figure out how to make that viewer work over the Jupyter web socket.
    • avoid opening any new ports
    • use Jupyter communication portal as much as possible
    • ImJoy
    • there may be work from ipyparaview we can use
  3. set up the synchronizable vtk.js window
  4. Abstract the API so it can easily accept any vtkRenderWindow object (VTK-Python or ParaView)
    • Deploy various Docker images that are ready to go with each accepted backend configured
  5. we can rebuild the concept of ipyparaview using work done in vtk.js to translate mouse events in vtk.js to the server and stream a screenshot.
    • All of the tracking/translation for camera movements in ipyparaview is implemented in vtk.js
  6. then we need to optimize how data is transferred when doing client-side rendering to make sure we are not resending the same data repeatedly
  7. then figure out how to let the client send data back to the server (like changing colormaps, etc.) and keeping all of those attributes in sync.
tjcorona commented 4 years ago

If you run into any trouble fixing issues in vtkJSONRenderWindowExporter I'd be happy to help.

banesullivan commented 4 years ago

One additional point: this current proposal would require server-side rendering no matter what: how that is conveyed to the client is different in the two modes (streaming a screenshot vs. synchronizing a scene graph with vtk.js). We should also explore ways to have pure client-side rendering without any vtkRenderWindow on the server.

An example of this would is PyVista's PlotterITK class which sends VTK data objects directly to the vtk.js renderer via itkwidgets on the client; see https://docs.pyvista.org/plotting/itk_plotting.html#example-plotting-with-itkwidgets. In order to implement pure client-side rendering, we would have to walk a fine line between developing a plotting API like what itkwidgets, PyVista, or Mayavi have done vs. a general way to only send data/scene to a vtk.js renderer. So this task may be best left for downstream projects and we only provide a way to stream data into a vtk.js scene with/without a server-side vtkRenderWindow instance.

jourdain commented 4 years ago

Hi @banesullivan,

Your summary is very good and I agree with your conclusion on the fact that a more robust solution should be taken in order to enable iPyParaView in a more production fashion.

Let me comments on some of your suggestions:

  1. I'm not sure about that part. WSLink is great outside of Jupyter for dealing with client/server communication but does not pretend to be anything more. Inside Jupyter, it seems we have something similar that could be used instead? Ideally if we can adapter Jupiter communication infrastructure to make it looks like the flexible wslink, that would be great. And by doing so, we could reuse a lot of ParaViewWeb code base. (Meaning applications: divvy, visualizer, lite...) WsLink could be used as a Jupiter adapter where it gains other communication backend. Then I'm not sure to understand what you mean by "proto for synchronization of client side rendering"? To my understanding itk-vtk-widget deals with data and offer a fully featured client side viewer. If your goal it to extend that viewer to allow it to either deal with data or rendering infrastructure on the the server side, I'm worry that it could increase its complexity beyond just a quick prototype. But in my opinion, if you want to enable such feature in itk-vtk-viewer as well as other viewers such as Glance, you should maybe extend that class instead. Kind of bringing what I'm doing here inside the vtkViewProxy API but with a more convenient API.

  2. This is indeed key and if you want to start with something small, you could look at a very basic vtk/pv web example app that is available here (same client but use either VTK or PV as python server). You could even strip down the Vue.js part and just do a plain JS proto that has nothing more than the 3D view and a slider to adjust the cone resolution. This example is relying on remote rendering.

  3. Is mostly relevant for small data that can be rendered with vtk.js. And if you figure out 2, that means you already support that part as we use that technic in PVW to enable local rendering. But depending on the time available, it could be nice to clean some things up here and there.

  4. This is already done as both VTK and ParaView use the same server and client side code.

  5. Yes indeed, but we might also need to optimize some stuff there too as I've noticed slowness with ParaView Lite (which sends camera) vs ParaView Visualizer (which sends mouse position). But my guess is that we just need to network debounce the push of the camera. So nothing major but some investigation should be made and streamlined via the exposed API of (1).

  6. This is already done by design especially if you enabled (2). One other optimization could be to slim down the json scene description to skip any prop that have not changed since the previous state. But that should be fairly small and won't offer much benefit.

  7. Well, if you did 2, that means you have a bi-directional channel that allow you to do RPC+Pub/Sub. Which means all of that became trivial.

In summary, the key part is (2) and to test/validate that, the smallest code would be the best. Pick the one you are more comfortable with such as itk-vtk-widget or the template example I pointed you to. But ideally we want to achieve RPC+Pub/Sub with binary attachment (wslink). Once we have that, we open the door to a more flexible solution where many of the pvw experience+code can be reused.

HTH,

Seb

jourdain commented 4 years ago

As I noticed the title just now, I would suggest to put code inside the vtk/web/*/ Python sub-directory so both VTK and ParaView can leverage some of that infrastructure code. Similar to what we do with the render window synchronization. But I would not necessary rebrand it. I would let ipyparaview and pyvista leverage the same code base to enable similar capabilities down the road with different backend.

thewtex commented 4 years ago

In summary, the key part is (2) and to test/validate that

@jourdain , this is solved by imjoy-rpc, which has robustly addressed the tricky issue of re-using the Jupyter connection. In addition, communicating via imjoy-rpc will provide

  1. JWT auth outside of a Jupyter context
  2. Zstd / Blosc / wasm compression for binary data for performance

Especially since @banesullivan is new to many of these technologies, to find an achievable path forward, it helps to start from working, proven technologies, and make incremental steps to enhance and factor out functionality for re-usability. E.g.,

  1. itk-vtk-viewer + wslink + VTK Python server
  2. itk-vtk-viewer + imjoy-rpc + VTK Python server
  3. itk-vtk-viewer + vtk.js Proxy, etc. + imjoy-rpc + VTK Python server
  4. itk-vtk-viewer + vtk.js Proxy, etc. + imjoy-rpc + VTK Python server + itk-jupyter-widgets
  5. itk-vtk-viewer + vtk.js Proxy, etc. + imjoy-rpc + ParaView Python server + itk-jupyter-widgets
  6. PVG + vtk.js Proxy, etc. + imjoy-rpc + ParaView Python server
jourdain commented 4 years ago

That make sense. And in some way wslink is not even required to try out. The only part that matter is the concept behind it. The tech is not relevant. If imjoy solve that over ws when used with a python server and over Jupiter comm when used inside Jupiter, then no question ask. Go for it.

Also, I was just providing some other pointers for other examples that was for me simpler. But what ever is simpler for Bane, should be his path. I’m fine either way.

On Wed, May 20, 2020 at 22:46 Matt McCormick notifications@github.com wrote:

In summary, the key part is (2) and to test/validate that

@jourdain https://github.com/jourdain , this is solved by imjoy-rpc https://www.npmjs.com/package/imjoy-rpc, which has robustly addressed the tricky issue of re-using the Jupyter connection. In addition, communicating via imjoy-rpc will provide

  1. JWT auth outside of a Jupyter context
  2. Zstd / Blosc / wasm compression for binary data for performance

Especially since @banesullivan https://github.com/banesullivan is new to many of these technologies, to find an achievable path forward, it helps to start from working, proven technologies, and make incremental steps to enhance and factor out functionality for re-usability. E.g.,

  1. itk-vtk-viewer + wslink
  2. itk-vtk-viewer + imjoy-rpc + VTK Python server
  3. itk-vtk-viewer + vtk.js Proxy, etc. + imjoy-rpc + VTK Python server
  4. itk-vtk-viewer + vtk.js Proxy, etc. + imjoy-rpc + VTK Python server
    • itk-jupyter-widgets
  5. itk-vtk-viewer + vtk.js Proxy, etc. + imjoy-rpc + ParaView Python server + itk-jupyter-widgets
  6. PVG + vtk.js Proxy, etc. + imjoy-rpc + ParaView Python server

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Kitware/resonanthpc/issues/16#issuecomment-631717083, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACH45SSYK55DILGSP3XUQDRSQ6RHANCNFSM4M6G4XVQ .

banesullivan commented 4 years ago

Everyone's input here is really awesome and has been incredibly insightful for me! @jourdain, I appreciate your detailed comments and @thewtex, thank you for clarifying a roadmap forward for me as I am definitely new/unfamiliar with much of this tech.

I am going to move forward on building step 1 that Matt has listed to familiarize myself with everything and I will try to report back as I can/as I have questions/progress.

banesullivan commented 4 years ago

@aashish24 asked me to revive this issue, touching on a few goals for the new ipyvtk package that will be this solution.

ipyvtk-simple is an early prototype of creating a Jupyter interface to VTK (effectively a simpler re-implementation of ipyparaview). ipyvtk will be a more polished tool.

Why Switch from ipyparaview?

ipyparaview provides an excellent proof of concept for streaming server-side 3D rendering to a client in the Jupyter environment. We plan to reimplement this toolkit leveraging existing software from Kitware's portfolio to generalize the tool for interfacing with any VTK-based rendering backend and leverage previous/proven work that has accomplished many parts of what will make this concept possible. Our goals/reasoning in creating a new software include:

jourdain commented 4 years ago

Are we still going to have a ipyparaview that will be based on ipyvtk?

thewtex commented 4 years ago

few goals for the new ipyvtk package that will be this solution.

All of these features are in progress in the itkwidgets package. There should not be an ipyvtk packages.

banesullivan commented 4 years ago

Are we still going to have a ipyparaview that will be based on ipyvtk?

My plan was to also have an ipyparaview subpackage based on ipyvtk that extends ipyvtk to handle more ParaView-specific features like time controls, a pipeline browser, etc.

All of these features are in progress in the itkwidgets package. There should not be an ipyvtk packages.

This is interesting... @thewtex, we may need to sync up and talk about the itkwidgets roadmap to see if my efforts would be best spent contributing to itkwidgets instead of working towards a new ipyvtk. Obviously, ipyvtk would be using much of the same underlying tech as itkwidgets but its goal would be to strip down the dependencies and be a general-purpose widget for any VTK-based software. Perhaps all of this is achievable with itkwidgets and its dependencies are well managed making that concern not as relevant?

tjcorona commented 4 years ago

Perhaps all of this is achievable with itkwidgets and its dependencies are well managed making that concern not as relevant?

We should definitely make sure we are not duplicating efforts (there's already enough to do!). Would it be possible to factor out the VTK components of itkwidgets and use it to seed an ipyvtk project? My understanding of itkwidgets is likely out of date, but I recall that VTK was an optional dependency. If that is still the case, we should map out a division of tasks and functionality that would result in smaller, more targeted projects that have a narrow focus on accomplishing a few things well (some lessons I am learning from wrestling a few antipatterns in VTK).

@thewtex, I see your buy-in to whatever design we agree upon as essential, given your experience in this field and how people would like to use it in their projects. Can you think of a way to simultaneously address the needs of the vtk-only crowd, the itk-only crowd, the paraview crowd and the folk who would like to interoperate between these libraries?

thewtex commented 4 years ago

This is interesting... @thewtex, we may need to sync up and talk about the itkwidgets roadmap to see if my efforts would be best spent contributing to itkwidgets instead of working towards a new ipyvtk.

@banesullivan Yes, I would be happy to sync up. Yes, as discussed in emails, everything you listed is on the roadmap and in progress in itkwidgets. Your goals are achievable with itkwidgets when working together.

We should definitely make sure we are not duplicating efforts (there's already enough to do!).

@tjcorona I completely agree.

Would it be possible to factor out the VTK components of itkwidgets and use it to seed an ipyvtk project? My understanding of itkwidgets is likely out of date, but I recall that VTK was an optional dependency. If that is still the case, we should map out a division of tasks and functionality that would result in smaller, more targeted projects that have a narrow focus on accomplishing a few things well (some lessons I am learning from wrestling a few antipatterns in VTK).

Perhaps it is my failure in communication, but itkwidgets has equally good support for VTK and ITK. In fact, there are more VTK than ITK examples:

https://github.com/InsightSoftwareConsortium/itkwidgets/tree/master/examples

The project is focused on two-way communication Python-JavaScript communication for spatial visualization that works well with Jupyter and the scientific Python ecosystem. This has required significant effort, and it should not be duplicated. The technical detail that vtk is not listed on the package dependencies (due to Python's dynamic capabilities, it is used whenever installed, though) is due to the fact that packages have not been consistently available on PyPI and conda-forge. With your work @tjcorona, that is being fixed, thankfully :smile: :pray:

Can you think of a way to simultaneously address the needs of the vtk-only crowd, the itk-only crowd, the paraview crowd and the folk who would like to interoperate between these libraries?

Yes, moving forward together with itkwidgets will simultaneously address the needs of the vtk-only crowd, the itk-only crowd, the paraview crowd, and the cool people that would like to interoperate between these libraries.

tjcorona commented 4 years ago

The technical detail that vtk is not listed on the package dependencies (due to Python's dynamic capabilities, it is used whenever installed, though) is due to the fact that packages have not been consistently available on PyPI and conda-forge.

I don't think its the absence of vtk as much as the presence of itk that is awkward. For someone who wants to go down the vtk-only path, itk is a confusing dependency. It is also confusing to need to use itkwidgets when the user has no intention of using itk. For these reasons (dependency and branding), I think it makes sense to consider refactoring core functionality into a base project that has fewer dependencies (if it is possible).

thewtex commented 4 years ago

I don't think its the absence of vtk as much as the presence of itk that is awkward. For someone who wants to go down the vtk-only path, itk is a confusing dependency.

@tjcorona I hear you, but I don't think the presence of itk is awkward. matplotlib, itk, numpy, scipy, ipywidgets ... are all package dependencies that are used internally to provide the functionality required. A user may have no intention of using matplotlib, and that is fine. When operating "vtk-only" there will be dependencies beyond the vtk package if we want to work with Jupyter and the scientific Python ecosystem in a meaningful way. The itk package has been released in cross-platform binaries bi-annually for three years, it is stable, and it is not a large, monolithic package like the vtk package -- it is split into smaller packages, and itkwidgets only depends on what is required.

The core functionality is not primary contained in either the itk or vtk Python packages -- this functionality is contained in itkwidgets, and most of the work is JavaScript-related. For branding purposes, the name should be itkwidgets to recognize this and support this role. We do not have, for example, a fork of vtk.js for itkwidgets. The ITK community supports and recognizes VTK, and it is reasonable to expect reciprocation.

jourdain commented 4 years ago

I don't think anyone wants to minimize the work that was put into itkwidgets that indeed benefit the whole Kitware stack (itk, vtk, paraview, ...).

I think where some are getting confused is how we can educate our community to think that they should use itkwidgets when they want to leverage itk, vtk or paraview inside a Jupyter environment. A natural implicit naming could possibly simplify that process. And right now, the naming seems to favor itk which make sense as it was the driver of the technology. But it also seems to confuse vtk or paraview users who have no clue what itk is.

For example, in the case of ParaViewWeb, we extracted the communication infrastructure to its own package so it could be used either in VTK, ParaView, ITK or any Python based server. Such technology was branded VTKWeb, ParaViewWeb and could very well be integrated inside ITK and branded ITKWeb if that was of interest. This provide a streamline communication to the community. Nobody care (and nobody should care) about wslink which only aims to provide a link over websocket.

Maybe what @thewtex is saying, is that itkwidgets could indeed be that base package to build ipyvtk and/or ipyparaview. And if any infrastructure code needs to be contributed, it should go inside itkwidgets while ipyvtk or ipyparaview could provide more dedicated UI and tools for the kind of work we do with ParaView and/or VTK.

@tjcorona would such approach make sense?

tjcorona commented 4 years ago

itkwidgets could indeed be that base package to build ipyvtk and/or ipyparaview. And if any infrastructure code needs to be contributed, it should go inside itkwidgets while ipyvtk or ipyparaview could provide more dedicated UI and tools for the kind of work we do with ParaView and/or VTK.

Sure, this solution would certainly help with the branding concerns we currently have.