search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
138 stars 51 forks source link

Filename extensions: Allow specifying wildcards #1584

Closed franzpoeschel closed 7 months ago

franzpoeschel commented 8 months ago

Close #1561

Diff: https://github.com/franzpoeschel/openPMD-api/compare/refactor-extract-bufferedactions..topic-wildcard-extension

Usage idea:

> bin/openpmd-pipe --infile ../samples/git-sample/3d-bp4/example-3d-bp4_%T.%E --outfile out.%E                                                              
Opening data source                                                                                                                                         
Opening data sink                                                                                                                                           
Traceback (most recent call last):                                                                                                                          
  File "/home/franzpoeschel/build/nbuild/bin/openpmd-pipe", line 17, in <module>                                                                            
    pipe.main()                                                                                                                                             
  File "/home/franzpoeschel/build/nbuild/lib/python3.11/site-packages/openpmd_api/pipe/__main__.py", line 366, in main                                      
    run_pipe.run()                                                                                                                                          
  File "/home/franzpoeschel/build/nbuild/lib/python3.11/site-packages/openpmd_api/pipe/__main__.py", line 211, in run                                       
    outseries = io.Series(self.outfile, io.Access.create,                                                                                                   
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                   
openpmd_api.openpmd_api_cxx.ErrorWrongAPIUsage: Wrong API usage: Unable to automatically determine filename extension. Please specify in some way.          
[~Series] An error occurred: fileBased output can not be written with no iterations.

> bin/openpmd-pipe --infile ../samples/git-sample/3d-bp4/example-3d-bp4_%T.%E --outfile out.%E --outconfig 'backend = "hdf5"'
Opening data source
Opening data sink
Opened input and output
Iteration 550 contains 9 meshes:
         B
         E
         e_all_chargeDensity
         e_all_energyDensity
         e_all_particleMomentumComponent
         i_all_chargeDensity
         i_all_energyDensity
         i_all_particleMomentumComponent
         picongpu_idProvider

Iteration 550 contains 2 particle species:
[…]

> ls 
out.h5

> bin/openpmd-ls out.%E                                  
openPMD series: out
openPMD standard: 1.1.0
openPMD extensions: 0

data author: unknown
data created: 2021-07-15 13:51:27 +0000 
data backend: HDF5
generating machine: unknown
generating software: PIConGPU (version: 0.6.0-dev)
generating software dependencies: unknown

number of iterations: 2 (groupBased)
  all iterations: 550 600 

number of meshes: 0 # this is a bug

number of particle species: 2
  all particle species:
    e
    i

> bin/openpmd-pipe --infile ../samples/git-sample/3d-bp4/example-3d-bp4_%T.%E --outfile out.%E --outconfig 'backend = "adios2"'
Opening data source
Opening data sink
Opened input and output
Iteration 550 contains 9 meshes:
         B
         E
         e_all_chargeDensity
         e_all_energyDensity
         e_all_particleMomentumComponent
         i_all_chargeDensity
         i_all_energyDensity
         i_all_particleMomentumComponent
         picongpu_idProvider

Iteration 550 contains 2 particle species:

> ls 
out.bp
out.h5

> bin/openpmd-ls out.%E            
An error occurred while opening the specified openPMD series!
Read Error in frontend 
Object type:    File
Error type:     Other
Further description:    Found ambiguous filename extensions on disk: '.h5' and '.bp'.

Todo:

PrometheusPi commented 6 months ago

Thanks for implementing that file ending wild card @franzpoeschel