search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
134 stars 51 forks source link

Examples: plotly express & holoviews #1182

Open ax3l opened 2 years ago

ax3l commented 2 years ago

With our new Pandas (& Dask) data frame APIs, we could add super rapid examples that use plotly express: https://plotly.com/python/plotly-express/ or holoviews: http://holoviews.org/user_guide/Tabular_Datasets.html

rhay-repo commented 2 years ago

Hi, I'm Reagan and I've been asked to contribute to an open-source project for my Software Development course. I'm interested in your team's project and interested in this issue in particular.

Could you tell me more about these examples you are hoping to produce with plotly express or holoviews that are mentioned in this issue? If I could help, I would like to be assigned this issue. Thank you.

ax3l commented 2 years ago

Hi @rhay-repo, nice to meet you!

That's fantastic, welcome to openPMD! :sparkles:

Organization

I am thinking about creating a new repo with cool jupyter notebook showcases (including https://mybinder.org startup). These notebooks could use openPMD-api and combine it with external projects like the two above.

Later on, we could use such notebooks as sections of tutorials.

What to Visualize

We have a couple of example data sets under https://github.com/openPMD/openPMD-example-datasets/ with which we can start.

Generally, we have particle data, which are effectively data frames (tables), and we have regular meshes (usually 2D-3D). A few default visualization that we need are show in this DASK use case I did recently: #958 https://github.com/dask/dask/issues/7307#issuecomment-815373939 https://github.com/openPMD/openPMD-api/pull/963#issuecomment-873350174

For particles, interactive 3d scatter plots, histograms/heatmaps over positions or position-momentum combinations and radar/polar charts of momentum components are generally very cool and useful for us as particle accelerator & laser-plasma physics people. I see those in https://plotly.com/python/plotly-express/#gallery

For holoviews, I would be curious about the same representations and how well interaction with large data works: http://holoviews.org/user_guide/Large_Data.html Maybe we can also build a testcase using our ADIOS SST support and their streaming data: http://holoviews.org/user_guide/Streaming_Data.html

What are your thoughts and ideas? We can store a lot of things in openPMD data sets :)

QA

For maintainability & QA, we could run the notebooks in CI to make sure we don't miss when we do breaking changes in openPMD-api or if breaking changes appear in third party projects that we interface with. This might not work with all plots, but we can try. A few technolgy routes like converting to Python scripts via nbconvert or running through nbterm exist.

rhay-repo commented 2 years ago

Thank you for including me in this discussion! This seems very cool! I worked with Plotly a couple years ago but I'd be new to HoloViews. Although the interactive HoloViews charts seem awesome and I'd be happy to try and utilize HoloViews.

For this project, I would be willing to experiment with both Plotly and Holoviews to see what each is capable of producing, report that back to you, and then pick the better of the two. Or if you decide to rule one of them out, that's great too.

I have cloned the raw datasets that you linked and will begin working with Plotly to start and try to replicate/mimic some of the default visualization you have done so far.

Question: Are you hoping to develop this visualization only for use as a one-time example/demonstration (with the example data sets)? or are you hoping to use this functionality repeatedly with new data input and visualize it for analytical purposes, etc.? I am trying to get a better grasp of your goal so that I can be as helpful as possible

bussmann commented 2 years ago

I think with some applications from laser and beam driven plasma accelerators one could aim for reusable, quasi-standard plots, e.g. electron density, phase space, energy histograms, field strengths, etc.

ax3l commented 2 years ago

Are you hoping to develop this visualization only for use as a one-time example/demonstration (with the example data sets)? or are you hoping to use this functionality repeatedly with new data input and visualize it for analytical purposes, etc.? I am trying to get a better grasp of your goal so that I can be as helpful as possible.

At the moment, I would like to show more use-cases how we can connect our data to modern visualization frameworks like these two. This would help domain-specific applications to pick them up and build dashboards.

For instance, we have a couple of general and laser-plasma driven default visualizations over here:

this uses mainly matplotlib for the final plot at the moment.

Similarly, we have a community project for visualizing accelerators and beams over here:

that also uses matplotlib as visualization framework.

If I remember correctly, plotly express and holoviews came up with @phyzicist, according to our discussions last week at a workshop. I would like to present some examples for him and other experimental physicists that like these two vis. frameworks and want to adopt our data APIs from openPMD.