Cannot read openPMD file

[eschnett]

I want to open an ADIOS-based openPMD file. I am using VisIt 3.1.4 on macOS 11.4, downloaded as a binary.

I have written the file with our application (the Einstein Toolkit), and I can read it there as well. Both bpls and openpmd-ls display the information in the file correctly, and they don't indicate any errors.

There are two issues:

(1) I cannot get VisIt to even open the file. The file selection box only lets me open files, but ADIOS bp file are stored as a directory. I tried pointing VisIt to one of the files in the directory, but this didn't work either, not even when I explicitly choose OpenPMD as file format.

(2) I created a *.visit file that points to the openPMD file (i.e. directory). I cannot open this file either; VisIt reports the error

VisIt could not read from the file "/Users/eschnett/simulations/brill-lindquist-write/output-0000/brill-lindquist-write/brill-lindquist-write.openpmd.visit".

The generated error message was:

There was an error opening /Users/eschnett/simulations/brill-lindquist-write/output-0000/brill-lindquist-write/brill-lindquist-write.openpmd.visit. It may be an invalid file.  VisIt tried using the following file format readers to open the file: OpenPMD

The following error(s) may be helpful in identifying the problem:
The selected database type was not correct for the given file.  The message from the database was Not an openPMD file: Error opening the file

I again explicitly chose OpenPMD as file format.

[biagas]

@ax3l Would you have any insight on this issue?

[eschnett]

I just ran into a similar issue with ParaView, which had better error messages. There, the problem was that the binary's ADIOS library was built without support for Blosc. Could there be a similar issue here?

[biagas]

Our OpenPMD reader has no dependencies on ADIOS, only HDF5. Not sure if that is important, I am not familiar with that reader's internals at this moment. Did you try opening as ADIOS (or ADIOS2) instead of OpenPMD?

Would you be willing to provide a minimal set of data for testing purposes?

[ax3l]

Hi and thanks for the report & ping!

The status of the VisIt reader is a direct implementation of the openPMD-standard 1.* against the HDF5 C libraries, first developed as a standalone plugin a while back (org repo) and recently updated for mainlining by us (cc @dpugmire).

The VisIt plugin was developed a while back, and since then we created a new C++/Python middleware library called openPMD-api that can be used for rapid implementations and supports ADIOS1/ADIOS2/HDF5/JSON data files with a unified interface.

In the last ParaView release (5.9) we mainlined a first plugin based on this library, which in about ~450 lines lines of parallel Python at the moment. We include this in the ParaView binary distribution and lack of Blosc is a good issue to fix in it, thanks for raising this! (Note that this is only a problem if you wrote compressed files though.) Raised now in a ticket and PRs opened.

I would love for us to "modernize" the VisIt plugin for ADIOS2 support as well. For VisIt builds, is there a way to either also write Python plugins in the same way and include an additional library build in the VisIt distributions?

openPMD-api is designed to work well with CMake super-builds, adds no external dependencies besides the ones you already have (MPI/HDF5/ADIOS2/Python, all individually controllable), can be used directly from C++ or from Python, and would reduce the lines of logic and file format binding code that we have to maintain in VisIt. We have a C++11 version, but generally moved forward to C++14 as a requirement now (release 0.13.0+). Does this look compatible with the VisIt toolchain requirements?

[eschnett]

@biagas I cannot open the *.bp directory directly. However, I can point a *.visit file to it, and I can then explicitly open it as ADIOS2 file. This recognizes the metadata correctly. Since openPMD uses a particular way to lay out datasets in the file, a lot of internals are exposed that shouldn't be, but I was able to find a dataset to display.

Drawing that dataset crashed the engine. This is using VisIt 3.1.4, both the viewer and the engine running locally on macOS.

I am happy to provide the file. I reduced the size to 1.3 MByte; is that sufficiently small, or should it be even smaller?

[brugger1]

Yes, 1.3MB is fine. If you would like to attach it to the issue, you will need to put it in a "zip" or "tar.gz" file.

[eschnett]

Here is the sample file: brill-lindquist-write.it00000000.bp.tar.gz

[aowen87]

@eschnett which variables are you trying to plot?

You can open your directory as a database by launching visit from the command line: /path/to/VisIt -o /path/to/brill-lindquist-write.it00000000.bp

We should really allow that through the GUI as well...

I don't see many variables available once opened. There is a mesh, which successfully plots, and there's a single variable that starts off as a blank name. Trying to plot that causes a crash. The blank name may have something to do with that.

[ax3l]

Adding more example BP4 files as a reference: https://github.com/openPMD/openPMD-example-datasets/

[brugger1]

I just tried opening this with VisIt 3.1.4 on quartz.llnl.gov and got the following output from the engine when I tried to plot the one variable in the file:

/usr/tmp/brugger/quartz/visitbuild/visit3.1.4/src/databases/ADIOS2/avtADIOS2FileFormat.C 63 isSST 0
/usr/tmp/brugger/quartz/visitbuild/visit3.1.4/src/databases/ADIOS2/avtADIOS2FileFormat.C 64 isWDM 0
/usr/tmp/brugger/quartz/visitbuild/visit3.1.4/src/databases/ADIOS2/avtADIOS2FileFormat.C 65 isHDF5 0
 Identifier has 1 variables
FLAVOR= 1
OPEN A BASIC READER
----------- ADIOS Base create interface for  /g/g17/brugger/Downloads/brill-lindquist-write.it00000000.bp -----------------
----------- ADIOS Base reader for /g/g17/brugger/Downloads/brill-lindquist-write.it00000000.bp with 1steps -----------------
----------- ADIOS Base return MTSD interface 
 Server populates metadata with 1 variables.
GetVar: /data/0/meshes/admbase_lapse_rl00/admbase_alp ts= 0
terminate called after throwing an instance of 'std::runtime_error'
  what():  ERROR: current ADIOS2 library didn't compile with Blosc, can't read Blosc compressed data, in call to Get

It mentions the lack of being compiled with Blosc support, which was mentioned earlier. It looks like we need to rebuild adios with Blosc.

[brugger1]

The Blosc library is available on GitHub at https://github.com/Blosc/c-blosc2. The most recent version is 2.0.4. It uses CMake for building, so hopefully it will be fairly easy to build.

We will need to add a --blosc option to build_visit. We will also need to add the library to the third-party repository and add it to the 3.2.0 release. We should then modify the adios build to use Blosc if available.

Once that is working, we can add the data attached to this ticket to our data directory and then add some tests to the test suite.

This can be added to the 3.2RC.

[brugger1]

Here is a checklist of things to do for this ticket.

• [x] Add the c-blosc2 tar file to the third-party repository.
• [x] Add the c-blosc2 tar file to the 3.2.0 release of the third-party repository.
• [x] Add a new module for building blosc to build_visit. It should be optional, controlled with --blosc.
• [x] Enhance the adios module for building with blosc if available.
• [ ] Build blosc and rebuild adios in third party install location on quartz.
• [ ] Add sample data attached earlier to this ticket to the data directory.
• [ ] Add tests using the new data.

[JustinPrivitera]

* [ ]  Build blosc and rebuild adios in third party install location on quartz.

I don't have access to quartz... will that be an issue?

[brugger1]

That shouldn't be an issue. I can do it for you when you're at that point. Also, we can get you access to additional machines soon.

[JustinPrivitera]

c-blosc seems to require a newer version of cmake CMake 3.16.3 or higher is required. You are running version 3.14.7 How should I proceed with this?

[brugger1]

How about 3.18.2, that's what Ascent uses.

[JustinPrivitera]

Maybe this is a novice question, but how can I make cmake use a newer version? I've been using the cmake in the third_party directory.

[markcmiller86]

Maybe this is a novice question, but how can I make cmake use a newer version? I've been using the cmake in the third_party directory.

If I understand your question @JustinPrivitera, you would first want to modify build_visit so that it is capable of building and installing CMake 3.18.2. It might be as simple as what @brugger1 wound up doing in this older PR from 2018.

[JustinPrivitera]

If I understand your question @JustinPrivitera, you would first want to modify build_visit so that it is capable of building and installing CMake 3.18.2. It might be as simple as what @brugger1 wound up doing in this older PR from 2018.

Thanks! I'll take a look next time I have a chance.

[JustinPrivitera]

I made some progress on this by editing the bv_cmake.sh file as directed by @markcmiller86. I went ahead and tried to build visit using the updated instructions to get the newest version of cmake. However, I get the following complaint from build_visit when it tries to install cmake:

Checking for cmake-3.18.2
Downloading cmake-3.18.2.tar.gz . . .
verifying SHA256 checksum 5d4e40fc775d3d828c72e5c45906b4d9b59003c9433ff1b36a1cb552bbd51d7e for cmake-3.18.2.tar.gz . . .
shasum -a 256 failed: looking for 5d4e40fc775d3d828c72e5c45906b4d9b59003c9433ff1b36a1cb552bbd51d7e got 
Download attempt failed: https://github.com/visit-dav/third-party/releases/download/v3.2.0/cmake-3.18.2.tar.gz

Here you can see what happens when I check the sha256sum:

bash-4.2$ shasum -a 256 cmake-3.18.2.tar.gz 
a87e329f2360da166933f261d7dcde064fa280cd6f7bf6c97a865c49de4b4e64  cmake-3.18.2.tar.gz

This is strange for two reasons: 1) https://cmake.org/files/v3.18/cmake-3.18.2-SHA-256.txt shows that whatever I got for the sha is very wrong, and that the value I put in bv_cmake.sh to check for is correct. 2) I've changed the sha I'm checking for in bv_cmake.sh to the sha I get from running shasum -a 256 on cmake-3.18.2.tar.gz 3 times now, each time remaking the build_visit script and rerunning it to redownload the same cmake-3.18.2.tar.gz, and each time the sha is completely different.

This is the function that controls the cmake version and where I put in the correct sha to check for:

function bv_cmake_info
{
    export CMAKE_URL=${CMAKE_URL:-"https://cmake.org/files/v3.18/"}
    export CMAKE_VERSION=${CMAKE_VERSION:-"3.18.2"}
    export CMAKE_FILE=${CMAKE_FILE:-"cmake-${CMAKE_VERSION}.tar.gz"}
    export CMAKE_BUILD_DIR=${CMAKE_BUILD_DIR:-"cmake-${CMAKE_VERSION}"}
    export CMAKE_MD5_CHECKSUM="7a882b3764f42981705286ac9daa29c2"
    export CMAKE_SHA256_CHECKSUM="5d4e40fc775d3d828c72e5c45906b4d9b59003c9433ff1b36a1cb552bbd51d7e"
}

And here is a link to the whole file: https://github.com/visit-dav/visit/blob/626a2c7799f79eb4b4c9e9d976ccdba4f3218776/src/tools/dev/scripts/bv_support/bv_cmake.sh#L99

I'm at a loss as to how to proceed. @brugger1 @markcmiller86 any guidance would be appreciated.

[JustinPrivitera]

It seems that when I download it directly from the cmake website https://cmake.org/files/v3.18/ it has the correct sha (as expected). I guess I could investigate how build_visit goes about downloading the tarballs, because it makes no sense to me why running build_visit would somehow manage to change the sha of the cmake tarball. Any help would be appreciated.

[cyrush]

@JustinPrivitera build visit will also look in the third-party repo, could it be that there we have cmake with a different sha?

[biagas]

@JustinPrivitera you can also comment out the checksums in bv_cmake.sh temporarily while you get the rest of the build working. I think once the cmake tarball is added to our third party repo, you can compute the checksum from that file in the repo.

[JustinPrivitera]

Well, here's why I experienced those issues: #17460

[JustinPrivitera]

@eschnett I've been working on this issue for some time now. I built adios and adios2 with c-blosc and have been able to open your provided file with adios2 (v2.5). However, all the data within appears to be zeros and errors are thrown:

ERROR: Invalid BP file detected. Variable index offset (0) <= PG index offset (0)
ERROR: File open failed: /usr/WS2/justin/blosc-land/brill-lindquist-write.it00000000.bp

Perhaps our version of adios2 is out of date? What version of adios did you use to write the file you attached?

[eschnett]

I probably used ADIOS2 2.7 to write the file. (Version 2.8 is current.)

[JustinPrivitera]

I upgraded VisIt to ADIOS2 2.7 (built with blosc) and was able to successfully open the file. This issue is nearly resolved. Thanks for your patience :)

[JustinPrivitera]

[eschnett]

This figures lists the dataset name as /data/0/meshes/admbase_lapse/rl00/admbase_alp. This does not look correct. openPMD would not return such a dataset name. Did you open the file as ADIOS2 file instead?

openPMD would call this dataset simply admbase_alp. The prefix that you see is due to openPMD's internal organization of the datasets in the file. openPMD can split variables across multiple datasets, and if so, the variable should be visualized as one.

The coordinates are also wrong. It might be that the axis ticks simply label vertices. openPMD stores coordinate information, and if that was used, the origin would be at the centre of the box.

[JustinPrivitera]

Did you open the file as ADIOS2 file instead?

Yes.

It seems we've had the wrong operating assumptions for resolving your issue. Kathleen suggested earlier in this conversation the following:

Our OpenPMD reader has no dependencies on ADIOS, only HDF5. Not sure if that is important, I am not familiar with that reader's internals at this moment. Did you try opening as ADIOS (or ADIOS2) instead of OpenPMD?

Eric tried to open the provided file and saw the following error:

I just tried opening this with VisIt 3.1.4 on quartz.llnl.gov and got the following output from the engine when I tried to plot the one variable in the file:

/usr/tmp/brugger/quartz/visitbuild/visit3.1.4/src/databases/ADIOS2/avtADIOS2FileFormat.C 63 isSST 0
/usr/tmp/brugger/quartz/visitbuild/visit3.1.4/src/databases/ADIOS2/avtADIOS2FileFormat.C 64 isWDM 0
/usr/tmp/brugger/quartz/visitbuild/visit3.1.4/src/databases/ADIOS2/avtADIOS2FileFormat.C 65 isHDF5 0
 Identifier has 1 variables
FLAVOR= 1
OPEN A BASIC READER
----------- ADIOS Base create interface for  /g/g17/brugger/Downloads/brill-lindquist-write.it00000000.bp -----------------
----------- ADIOS Base reader for /g/g17/brugger/Downloads/brill-lindquist-write.it00000000.bp with 1steps -----------------
----------- ADIOS Base return MTSD interface 
 Server populates metadata with 1 variables.
GetVar: /data/0/meshes/admbase_lapse_rl00/admbase_alp ts= 0
terminate called after throwing an instance of 'std::runtime_error'
  what():  ERROR: current ADIOS2 library didn't compile with Blosc, can't read Blosc compressed data, in call to Get

It mentions the lack of being compiled with Blosc support, which was mentioned earlier. It looks like we need to rebuild adios with Blosc.

Hence our decision was to upgrade ADIOS by compiling it with c-blosc, and in the process, upgrading ADIOS2 to a newer version. It sounds like what you are looking for, however, is an upgrade to our openPMD reader, so that it depends on ADIOS, which depends on c-blosc. Is that correct?

If so, I or another team member could get started on that, but it could be some time before it is completed.

[biagas]

It sounds like what you are looking for, however, is an upgrade to our openPMD reader, so that it depends on ADIOS, which depends on c-blosc. Is that correct?

Based on @ax3l's comment above I believe an upgrade to our openPMD reader is indeed needed, perhaps utilizing the openPMD-api as suggested.

[dpugmire]

@eschnett I'm trying to use VisIt to open some openPMD Adios files. Is there a way to get these files into VisIt? It's not clear to me from the conversation if a solution has been found.

[biagas]

@dpugmire VisIt's openPMD reader needs an update in order to read those files. It's still an open issue.

[eschnett]

@dpugmire As @biagas says, reading openPMD files this is not (yet) possible.

[BenWibking]

Is this planned for 3.4.1?

Due to the much higher performance of ADIOS2 on Frontier compared to HDF5 (about 2 orders of magnitude), we've switched to openPMD/ADIOS2 outputs for our runs there, and we'd definitely like the ability to read those files with VisIt there.

[JustinPrivitera]

Hello. We will discuss this at the next team meeting and see if we will be able to get it in 3.4.1 or 3.4.2. 3.4.1 is probably coming pretty soon to fix some issues with 3.4.0.

[BenWibking]

I actually cannot open the example dataset provided above using the official macOS binary build of VisIt 3.3.3. I get this error:

ERROR: Invalid BP file detected. Variable index offset (0) <= PG index offset (0)
ERROR: File open failed: /Users/benwibking/brill-lindquist-write.it00000000.bp

[BenWibking]

I am able to open the file using 3.3.3 on Windows.

I also noticed that on Windows, it says the reader is ADIOS2, whereas on Mac, it's ADIOS. Is it possible that the ADIOS2 reader was not compiled with the macOS binary, and it instead has an older ADIOS1 reader?

[eschnett]

There are two different (widely used) ADIOS2 data formats, ADIOS2_BP4 and ADIOS2_BP5. if you use the latter with an older library you only get an unintelligible error message. it is unfortunate that the respective file format version number check is lacking in clarity.

If so then you have two options. Either regenerate the file with ADIOS2_BP4, or update to a newer library (either ADIOS2 or openPMD, I don't recall which, maybe both).

[BenWibking]

There are two different (widely used) ADIOS2 data formats, ADIOS2_BP4 and ADIOS2_BP5. if you use the latter with an older library you only get an unintelligible error message. it is unfortunate that the respective file format version number check is lacking in clarity.

If so then you have two options. Either regenerate the file with ADIOS2_BP4, or update to a newer library (either ADIOS2 or openPMD, I don't recall which, maybe both).

Since this PR (https://github.com/visit-dav/visit/pull/17622) was merged into VisIt, I don't see how that could be the case for the issue I'm seeing. Can you elaborate?

@eschnett I used the "brill_lindquist_write" BP file you uploaded and VisIt 3.3.3 on macOS. What versions of VisIt on macOS are you able to open it with?

[JustinPrivitera]

Since this PR (https://github.com/visit-dav/visit/pull/17622) was merged into VisIt, I don't see how that could be the case for the issue I'm seeing. Can you elaborate?

I don't think #17622 was merged, but #17653 was.

[BenWibking]

I don't think #17622 was merged, but #17653 was.

Ah, I missed that. Thanks for pointing to the right PR.

Is there a way to check the version of ADIOS that was included in the binary distribution? I tried this:

$ pwd
/Users/benwibking/Downloads/visit3_3_3.darwin-x86_64/3.3.3/darwin-i386/plugins/databases
$ otool -L libEADIOSDatabase_ser.dylib
libEADIOSDatabase_ser.dylib:
    @rpath/plugins/databases/libEADIOSDatabase_ser.dylib (compatibility version 0.0.0, current version 0.0.0)

[BenWibking]

With -debug 5, I see:

Loaded full database plugin ADIOS version 1.0

and later:

Reading file list from text file "/Users/benwibking/adios_database.visit"
Exception: (InvalidDBTypeException) /Users/miller86/visit/visit/33rc/release/build-mb-3.3.3-darwin-12.6-x86_64-release/visit/src/\
databases/ADIOS/ADIOSFileObject.C, line 155: The selected database type was not correct for the given file.  The message from the\
 database was /Users/benwibking/brill-lindquist-write.it00000000.bp
catch(VisItException) /Users/miller86/visit/visit/33rc/release/build-mb-3.3.3-darwin-12.6-x86_64-release/visit/src/databases/ADIO\
S/avtADIOSFileFormat.C:114

There is no ADIOS2 database plugin listed. I think this shows the VisIt 3.3.3 macOS build is missing the ADIOS2 database plugin.

[BenWibking]

With VisIt 3.2.x, I get the blosc error mentioned above.

On macOS, there appears to be no version of VisIt that can open brill-lindquist-write.it00000000.bp.

[eschnett]

If the ADIOS library file name does not contain the digit 2 then you are probably still using ADIOS 1.