Distinguish BP versions in file endings

[franzpoeschel]

Is your feature request related to a problem? If so, please describe. BP3 and BP4 are already used in production, BP5 is coming up. The formats are incompatible with each other, but we use the same endings for them so far. BP5 is not going to replace BP4, so we should go for a solution that will let people use both formats side by side without too much confusion.

Describe the solution you'd like Instead of simData.bp, write files like simData.bp3, simData.bp4, simData.bp5. Optionally even enforce version numbers if users don't specify them, e.g. Series("simData.bp", Access::CREATE) might create simData.bp4. Only do this if we find a solution that does not break existing workflows.

Describe alternatives you've considered Automatic version detection in ADIOS2 if using the generic ending .bp Being explicit about versions is still good

[pnorbert]

There are two differences between BP4 and BP5 directory content:

• BP5 has an mmd.0 file in it. This is created at open, filled in first step and maybe get more content later.
• md.idx byte 37 is the BP version - this is written with the first step (not at open)

  $ od -An -j 37 -N 1 -tu1 globalArray.bp/md.idx
  5

[franzpoeschel]

Thank you for the info. So we can use some form of automatic detection as long as there is no clear file ending.

[pnorbert]

Also, using FileStream as engine for reading makes adios check for the type of data, and launches BP4 or BP5.

[franzpoeschel]

Ah, very good So we can use that as default in reading procedures and not worry much about BP versions

[pnorbert]

Well, for 4 and 5, not for BP3

[franzpoeschel]

That's probably ok, since noone is really using BP3 anymore anyway

[ax3l]

Likely, and if they do we have a manual switch for it (after it crashes for them): OPENPMD_BP_BACKEND https://openpmd-api.readthedocs.io/en/0.14.4/backends/adios1.html

[ax3l]

Being able to use .bp3, .bp4 and .bp5 explicitly would nonetheless be cool/explicit/transparent, as we discussed last week. We might need to settle for .bp (BP3), .bp4 and .bp5 in the end, if I remember the ADIOS1 conventions correctly.

[franzpoeschel]

OPENPMD_BP_BACKEND distinguishes ADIOS1 and ADIOS2, not BP3 and BP4

We might need to settle for .bp (BP3), .bp4 and .bp5 in the end, if I remember the ADIOS1 conventions correctly.

You remember correctly, I've removed the .bp3 file ending again since it does not work. Does not matter too much, the important thing is to properly distinguish BP4 and BP5. (Also BP3 is relatively easy to visually spot because it's a file, not a folder)