openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
139 stars 51 forks source link

JSON/TOML arguments for all Series constructor arguments #1685

Open franzpoeschel opened 1 week ago

franzpoeschel commented 1 week ago

Is your feature request related to a problem? If so, please describe.

1584 introduced wildcards for filenames. This comes in handy, since it allows writing the main code independent from the backend which is then specified via JSON/TOML:

// main.cpp
int main() 
{
    // ...
    Series output("simData.%E", Access::CREATE, "@config.toml");
    // ...
}
# config.toml
backend = "adios2"
iteration_encoding = "variable_based"
# --> a file named simData.bp5 will be written

Or alternatively:

# config.toml
backend = "hdf5"
iteration_encoding = "group_based"

This makes it very easy to write openPMD-based output routines that can be configured at runtime by the user, without needing to write a parser for command line arguments or similar.

Problem: This falls short when trying to use file-based iteration encoding since JSON/TOML exposes no way to specify the %T expansion pattern. The logical consequence imo is to allow specifying everything in JSON/TOML that could be specified in the constructor.

Describe the solution you'd like

Series(
    std::string const& filepath, // default, can be overridden from JSON/TOML
    Access at, // default, can be overridden from JSON/TOML
    std::string const &json_toml);
file_base_name = "simData"
file_infix = "%T"
# --> will be put together as "simData_%T.%E"
# "infix" is optional, will default as %06T in file-based encoding, as empty otherwise
file_base_name = "simData"
file_extension = "bp"
iteration_encoding = "file_based"
# --> will be put together as "simData_%06T.bp"
franzpoeschel commented 1 week ago

Alternative easier fix: In group/variable-based encoding, erase _%06T and similar patterns from the specified filename. Add a JSON/TOML config option to suppress this behavior.