no rendering using interactive openPMD-viewer

BenWibking commented 6 months ago

I am attempting to use the default interactive notebook to visualize a series of BP5 outputs. It recognizes the names of the components, but nothing shows up. In the console, it shows:

[I 2023-12-14 15:18:13.652 JupyterNotebookApp] 302 GET /tree/openPMD-visualization.ipynb?token=[secret] (1adbbb045dda4c4e9fc49c3a014e6e57@::1) 4.07ms
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[W 2023-12-14 15:18:15.366 ServerApp] Notebook openPMD-visualization.ipynb is not trusted
[I 2023-12-14 15:18:15.521 ServerApp] Kernel started: 22f6f31c-47fe-4adb-8800-a3b61c452b54
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 2023-12-14 15:18:15.812 ServerApp] Connecting to kernel 22f6f31c-47fe-4adb-8800-a3b61c452b54.
[AbstractIOHandlerImpl] IO Task AVAILABLE_CHUNKS failed with exception. Clearing IO queue and passing on the exception.

Each time I click "Refresh now!", it prints a new line:

[AbstractIOHandlerImpl] IO Task AVAILABLE_CHUNKS failed with exception. Clearing IO queue and passing on the exception.

ax3l commented 6 months ago

Hi @BenWibking ,

Thanks a lot for the report! I moved it over to openPMD-api, because the backend issue in openPMD-viewer is rooted here.

Can you provide us some deetails how openPMD-api was installed for your Jupyter/Python/openPMD-viewer stack?

Package manager or other method details
Version of openPMD-api and openPMD-viewer

Can you print the ls structure and the exact Python commands you use to read?

ax3l commented 6 months ago

@BenWibking, for testing your data set, can you do produce a comparable output with .h5?

The .h5 files, you can run through:

openPMD-validator and
open with openPMD-api and openPMD-viewer.

Does the validator show any errors or warnings? Does openPMD-api and openPMD-viewer consume the HDF5 (or alternatively, .bp4) files well?

BenWibking commented 6 months ago

For this case, I installed openPMD-api and openPMD-viewer with conda as follows:

conda create -n openpmd -c conda-forge openpmd-api
conda install -c conda-forge openpmd-viewer openpmd-api
conda install -c conda-forge openpmd-validator

>>> import openpmd_api
>>> openpmd_api.__version__
'0.15.2'
>>> import openpmd_viewer
>>> openpmd_viewer.__version__
'1.9.0'

Then I used openPMD_notebook to create the notebook, and then changed the path in cell 2.

Conda info:

(openpmd) ➜  tests git:(development) ✗ conda info

     active environment : openpmd
    active env location : /opt/homebrew/Caskroom/miniconda/base/envs/openpmd
            shell level : 2
       user config file : /Users/benwibking/.condarc
 populated config files :
          conda version : 23.7.4
    conda-build version : not installed
         python version : 3.11.6.final.0
       virtual packages : __archspec=1=arm64
                          __osx=14.2=0
                          __unix=0=0
       base environment : /opt/homebrew/Caskroom/miniconda/base  (writable)
      conda av data dir : /opt/homebrew/Caskroom/miniconda/base/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/homebrew/Caskroom/miniconda/base/pkgs
                          /Users/benwibking/.conda/pkgs
       envs directories : /opt/homebrew/Caskroom/miniconda/base/envs
                          /Users/benwibking/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.7.4 requests/2.29.0 CPython/3.11.6 Darwin/23.2.0 OSX/14.2
                UID:GID : 501:20
             netrc file : None
           offline mode : False

@BenWibking, for testing your data set, can you do produce a comparable output with .h5?

The .h5 files, you can run through:

openPMD-validator and

open with openPMD-api and openPMD-viewer.

Does the validator show any errors or warnings? Does openPMD-api and openPMD-viewer consume the HDF5 (or alternatively, .bp4) files well?

The validator shows:

(openpmd) ➜  tests git:(development) ✗ openPMD_check_h5 -i hdf5/plt00000.h5
Warning: Attribute author (recommended) does NOT exist in `/`!
Found 1 iteration(s)
Iteration 0 : found 6 meshes
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/bin/openPMD_check_h5", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 931, in main
    result_array = check_file(file_name, verbose, force_extension_pic)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 924, in check_file
    result_array += check_iterations(f, verbose, extensionStates)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 495, in check_iterations
    result_array += check_meshes(f, iteration, v, extensionStates)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 603, in check_meshes
    result_array += test_record(f[full_meshes_path], field_name)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 159, in test_record
    %(r.name) )
      ^^^^^^
AttributeError: 'str' object has no attribute 'name'

and openPMD-viewer shows the same error as before:

[AbstractIOHandlerImpl] IO Task AVAILABLE_CHUNKS failed with exception. Clearing IO queue and passing on the exception.

Here are the files in both HDF5 and BP5 formats: quokka_openpmd.tar.gz

BenWibking commented 6 months ago

This script shows that the data appears to be there, so I'm not sure what's going on...

import openpmd_api as io
import numpy as np
import matplotlib.pyplot as plt

filename = "bp5/plt00758.bp"
#filename = "hdf5/plt00758.h5"
series = io.Series(filename, io.Access.read_only)
my_iter = series.iterations[list(series.iterations)[0]]
mesh = my_iter.meshes

def readComp(varname):
    comp = mesh[varname][io.Mesh_Record_Component.SCALAR][()]
    series.flush()
    return comp

# read gasDensity
density = readComp("gasDensity")
plt.clf()
plt.figure(figsize=(4,4))
im = plt.imshow(density, origin='lower')
plt.colorbar(im)
plt.tight_layout()
plt.savefig("gasDensity.png", dpi=150)

gasDensity

BenWibking commented 6 months ago

If I change line 159 of check_h5.py to print r instead of r.name, I get:

$ openPMD_check_h5 -i hdf5/plt00000.h5 
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:147: SyntaxWarning: invalid escape sequence '\w'
  regEx = re.compile("^\w+$") # Python3 only: re.ASCII
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:407: SyntaxWarning: invalid escape sequence '\.'
  result_array += test_attr(f, v, "required", "openPMD", np.string_, "^[0-9]+\.[0-9]+\.[0-9]+$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:409: SyntaxWarning: invalid escape sequence '\/'
  result_array += test_attr(f, v, "required", "basePath", np.string_, "^\/data\/\%T\/$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:414: SyntaxWarning: invalid escape sequence '\/'
  result_array += test_attr(f, v, "optional", "meshesPath", np.string_, "^.*\/$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:415: SyntaxWarning: invalid escape sequence '\/'
  result_array += test_attr(f, v, "optional", "particlesPath", np.string_, "^.*\/$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:431: SyntaxWarning: invalid escape sequence '\+'
  "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [\+|-][0-9]{4}$")
Warning: Attribute author (recommended) does NOT exist in `/`!
Found 1 iteration(s)
Iteration 0 : found 6 meshes
Error: Record x-GasMomentum is NOT named properly (a-Z0-9_)!
Error: Record y-GasMomentum is NOT named properly (a-Z0-9_)!
Error: Record z-GasMomentum is NOT named properly (a-Z0-9_)!
Iteration 0 : found 0 particle species
Result: 3 Errors and 1 Warnings.

BenWibking commented 6 months ago

I realized I was also missing axisLabels for all of the coordinate dimensions, but the validator didn't catch that. However, that doesn't fix the problem with openPMD-viewer.

BenWibking commented 6 months ago

With fixed iteration encoding, fixed axisLabels, and fixed component names: quokka_openpmd_fixed.tar.gz

Still nothing appears in openPMD-viewer.

franzpoeschel commented 6 months ago

I can reproduce this locally with your dataset. Unfortunately, the openPMD-viewer seems to swallow the original exception, so it is not obvious where the error originally stems from. I'll have a look.

franzpoeschel commented 6 months ago

The problem is that your files for some reason encode the wrong iteration encoding:

> bpls -al bp5/plt00000.bp/ -e '/iterationEncoding'
  string    /iterationEncoding                                 attr   = "groupBased"

As a consequence, the available_chunks() call does not know that it needs to open the file before accessing it. How did you create the file? Was this originally group-based output and you renamed it as file-based?

Aside: In https://github.com/openPMD/openPMD-api/pull/1368/files, we removed two warnings when the encoded iteration encoding did not match the filenames. We should reinstate the first of the two removed warnings. I'll prepare a PR.

franzpoeschel commented 6 months ago

With #1573:

> python
Python 3.10.12 (main, Jun  6 2023, 22:43:10) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import openpmd_api as io
>>> s = io.Series("bp5/plt%T.bp", io.Access.read_only)
Series constructor called with iteration regex '%T' suggests loading a time series with fileBased iteration encoding. Loaded file is groupBased. Will ignore the encoding stated in the file and continue treating this as file-based. Depending on what data the opened files actually contain, this might not yield correct results.
Series constructor called with iteration regex '%T' suggests loading a time series with fileBased iteration encoding. Loaded file is groupBased. Will ignore the encoding stated in the file and continue treating this as file-based. Depending on what data the opened files actually contain, this might not yield correct results.
>>> s.iterations[0].meshes["gasDensity"][io.Record_Component.SCALAR].available_chunks()
[<openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>]

Fixing the iteration encoding in your files should still be the preferred solution. Since ADIOS2 does not really support editing files once written, this is only really possible by copying/recreating the file. openpmd-pipe can help you with that (installed along with openPMD-api): openpmd-pipe --infile bp5/plt%T.bp --outfile fixed_%T.bp.

BenWibking commented 6 months ago

With fixed iteration encoding, fixed axisLabels, and fixed component names: quokka_openpmd_fixed.tar.gz

Still nothing appears in openPMD-viewer.

I tried to fix the iteration encoding issue in these files. The exception no longer appears in the console, but I still don't see any rendering of the data in openPMD-viewer. Can you reproduce this?

franzpoeschel commented 6 months ago

Ah, yep, should have read more carefully. The initial bug was something that needs to be fixed in the openPMD-api anyway, but the remaining issue seems to be an unrelated issue in the openPMD-viewer. I'll first ask @ax3l if he has an idea what's going on, I'm not very familiar with how the openPMD-viewer works.

What I see is just the GUI of the Viewer, without any actual output, I assume it's the same on your end? grafik

BenWibking commented 6 months ago

What I see is just the GUI of the Viewer, without any actual output, I assume it's the same on your end?

Yeah, that's exactly what I see as well.

franzpoeschel commented 6 months ago

Huh I tried this again, and changing %matplotlib notebook to %matplotlib widget did the trick for me. Wonder what's the meaning of that.. grafik

BenWibking commented 6 months ago

It works!

Maybe that could be changed to the default? When I loaded it with notebook first, then changed it to widget, it gives an error that at first led me to think that widget wouldn't work at all:

However, if I change it to widget before running the first cell, it works fine and produces the same result you show above.

franzpoeschel commented 6 months ago

Maybe that could be changed to the default? When I loaded it with notebook first, then changed it to widget, it gives an error that at first led me to think that widget wouldn't work at all:

Let me be honest with you here: I have no idea what either of these two options do, I was just poking things until they worked.

But these observations might be interesting to @RemiLehe (maintainer of the openPMD-viewer) and @ax3l, as this sounds at least like a UX issue of the openPMD-viewer, if not a bug.

BenWibking commented 6 months ago

According to the matplotlib docs, it looks like %matplotlib notebook only works for old versions of notebooks, whereas %matplotlib widget works on all modern versions of either Jupyter Notebook or Jupyterlab: https://matplotlib.org/stable/users/explain/figure/interactive.html#jupyter-notebooks-jupyterlab.

openPMD / openPMD-api

no rendering using interactive openPMD-viewer #1572