Python API does not work for 2.9.2

[hr87]

Describe the bug The python API in version 2.9.2 fails to return any data. I get empty lists for both attributes and variables. The file has been writen with Fortran API 2.9.2, bpls shows attributes with values. It worked well with 2.8.2.

To Reproduce

1. Write ADIOS file containing attributes
2. Try to obtain attributes with Python API

(code) user@machine:~/.../> bpls -Al code.out
  double   constants/avogadro               attr   = 6.02214e+23
  double   constants/boltzmann              attr   = 1.60218e-19
  double   constants/electric               attr   = 8.85419e-12
  double   constants/electron_mass          attr   = 9.10938e-31
  double   constants/euler                  attr   = 0.577216
  double   constants/fine_structure         attr   = 0.00729735
  double   constants/gas_constant           attr   = 96485.3
  double   constants/magnetic               attr   = 1.25664e-06
  double   constants/natural_charge         attr   = 1.60218e-19
  double   constants/neutron_mass           attr   = 1.67493e-27
  double   constants/pi                     attr   = 3.14159
  double   constants/planck                 attr   = 6.62607e-34
  double   constants/planck_bar             attr   = 1.05457e-34
  double   constants/proton_mass            attr   = 1.67262e-27
  double   constants/radiation              attr   = 13.7202
  double   constants/speed_light            attr   = 2.99792e+08
  double   constants/stefan_boltzmann       attr   = 1.0283e+09
(code) user@machine:~/.../> bpls -l code.out
  double   mesh/cell_centers                      357*{130}
  double   mesh/coordinate_jacobians              357*{130}
  double   mesh/face_jacobians                    357*{129}
  double   mesh/face_positions                    357*{129}
  double   mesh/face_speeds                       357*{131}
  double   mesh/mesh_jacobians                    357*{130}
(code) user@machine:~/.../> bpls --version
  2.9.2
(code) user@machine:~/.../> ipython
Python 3.11.2 (main, Mar 27 2023, 23:42:44) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.10.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import code.adios_helper

In [2]: import adios2

In [3]: handle = adios2.open('code.out', 'r')

In [4]: handle.available_attributes()
Out[4]: {}

In [5]: adios2.__version__
Out[5]: '2.9.2'

In [6]: handle.available_variables()
Out[6]: {}

Expected behavior Python should return variables and attributes found in the adios file

Env: OS:

Transient hostname: darwin-fe2.lanl.gov
         Icon name: computer-server
           Chassis: server
  Operating System: Red Hat Enterprise Linux 8.8 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8::baseos
            Kernel: Linux 4.18.0-477.15.1.el8_8.x86_64
      Architecture: x86-64

Code:

  Compiled gfortran 12.2.0 on darwin-fe2.lanl.gov
  MPI version: Open MPI v4.1.5, package: Open MPI dor@cn204 Distribution, ident: 4.1.5, repo rev: v4.1.5, Feb 23, 2023
  Petsc version: 3.20.1
  ADIOS version: 2.9.2

Additional context Worked with Adios 2.8.2

[eisenhauer]

This is a circumstance where unfortunately you have to read the release notes for 2.8 and 2.9 to infer what is going on. In order to address metadata scalability issues the new BP5 engine (which was not the default in 2.8), more strictly separated the two read "modes" in ADIOS which were only informally separated in the file engines before. In particular 2.8 introduced ReadRandomAccess mode, in which the entire set of metadata in the input file is read upon Open() (this was the default behaviour in BP3/4) and then individual data items are selected with SetStepSelection before Get(). However by default if you simple do Read mode, BP5 assumes the much more efficient version where it loads metadata on a per-timestep basis on BeginStep(). 2.9 made BP5 the default file engine. If using the C++ bindings there is a helpful exception that is raised in this circumstance, but unfortunately the python bindings swallow that exception and it gets missed. The solution is to either explicitly specify the BP4 engine, change your open to 'rra' mode, or to change the code to look for variables after BeginStep(). Which of these is most appropriate depends upon your situation.

The new BP5 behavior mirrors standard behavior in ADIOS streaming engines. The ambiguity in file engines was an unfortunate historical artifact.

Also see issues #3831 and #3843.

[hr87]

The fix seems easy enough, but it should be better documents and communicated.

I do not see anything mentioned here, that implies this: https://github.com/ornladios/ADIOS2/releases/tag/v2.9.0. This should go in a section like Breaking changes.

The documentation for Python does not mention random access at all. There is also no mention of this limitation in the documentation for these functions: https://adios2.readthedocs.io/en/v2.9.2/api_high/api_high.html#adios2.File.available_variables

[eisenhauer]

Community contributions are welcome...

[pnorbert]

Duly noted now in the release notes. Thanks for the feedback