Read_attribute broken with numpy > 2.0.0 on redhat and 2.1.1 on ubuntu

[jorgensd]

I have the following minimal script that writes a numpy array as an attribute in adios2, and in turn read it into the code again:

import adios2.bindings as adios2
from mpi4py import MPI
import numpy as np

engine = "BP5"
filename = "test.bp"

adios = adios2.ADIOS(MPI.COMM_WORLD)

io = adios.DeclareIO("writer")

out_data = np.array([13, 11, 7, 5], dtype=np.int32)

# Write array to file as attribute
io.SetEngine(engine)
adios_file = io.Open(str(filename), adios2.Mode.Write, MPI.COMM_WORLD)
adios_file.BeginStep()
io.DefineAttribute("data", out_data)
adios_file.PerformPuts()
adios_file.EndStep()
adios_file.Close()
adios.RemoveIO("writer")

# Read array from file as attribute
io = adios.DeclareIO("reader")
io.SetEngine(engine)
adios_file = io.Open(str(filename), adios2.Mode.Read, MPI.COMM_WORLD)
adios_file.BeginStep()
attr = io.InquireAttribute("data")
in_data = attr.Data()
adios_file.EndStep()
adios_file.Close()
adios.RemoveIO("reader")

np.testing.assert_array_equal(out_data, in_data)

This works fine with rockylinux 9:

FROM rockylinux/rockylinux:9

ARG ADIOS2_VERSION=v2.10.1

ARG NUMPY_VERSION=1.26.4
ARG MPICH_VERSION=4.2.2
WORKDIR /tmp

RUN dnf -y update && \
    dnf install -y dnf-plugins-core && \
    dnf config-manager --set-enabled crb && \
    dnf install -y epel-release && \
    dnf -y install \
    cmake \
    gcc \
    gcc-c++ \
    gcc-gfortran \
    pugixml-devel \
    python3 \
    python3-devel \
    python3-pip \
    spdlog-devel \
    # Utility
    git \
    # Only in crb set
    ninja-build \
    openblas-devel && \
    dnf -y clean all && \
    rm -rf /var/cache

# Build MPICH (see https://github.com/pmodels/mpich/issues/5811)
RUN curl -L -O https://www.mpich.org/static/downloads/${MPICH_VERSION}/mpich-${MPICH_VERSION}.tar.gz && \
    tar -xf mpich-${MPICH_VERSION}.tar.gz && \
    cd mpich-${MPICH_VERSION} && \
    FCFLAGS=-fallow-argument-mismatch FFLAGS=-fallow-argument-mismatch ./configure --with-device=ch4:ofi --enable-shared --prefix=/usr/local && \
    make -j 4 install && \
    rm -rf /tmp/*

# Install numpy and Mpi4py
RUN python3 -m pip install numpy==${NUMPY_VERSION} mpi4py

# BUILD ADIOS2
RUN git clone --branch=${ADIOS2_VERSION} --single-branch https://github.com/ornladios/ADIOS2.git adios2 && \
    cmake -G Ninja -DADIOS2_USE_HDF5=off -DADIOS2_USE_Python=on -DADIOS2_USE_Fortran=off -DBUILD_TESTING=off -DADIOS2_BUILD_EXAMPLES=off -DADIOS2_USE_ZeroMQ=off -B build-dir-adios2 -S ./adios2 && \
    cmake -G Ninja -B build-dir-adios2 -DCMAKE_BUILD_TYPE="Release" -S ./adios2/ && \
    cmake --build build-dir-adios2 --parallel 2 && \
    cmake --install build-dir-adios2

WORKDIR /src/test/
COPY mwe.py mwe.py
RUN python3 mwe.py

RUN bpls -a -l test.bp

which you can build with:

 docker build -t adios2_image --build-arg="NUMPY_VERSION=1.26.4" . --progress=plain

However, using numpy 2.0.1, one gets

 docker build -t adios2_image --build-arg="NUMPY_VERSION=2.0.1" . --progress=plain

with the following traceback

> [ 9/10] RUN python3 mwe.py:
1.777   File "/usr/local/lib64/python3.9/site-packages/numpy/testing/_private/utils.py", line 885, in assert_array_compare
1.777     raise AssertionError(msg)
1.777 AssertionError: 
1.777 Arrays are not equal
1.777 
1.777 Mismatched elements: 3 / 4 (75%)
1.777 Max absolute difference among violations: 8
1.777 Max relative difference among violations: 0.61538462
1.777  ACTUAL: array([13, 11,  7,  5], dtype=int32)
1.777  DESIRED: array([13, 13, 13, 13], dtype=int32)
------
Dockerfile:55
--------------------
  53 |     WORKDIR /src/test/
  54 |     COPY mwe.py mwe.py
  55 | >>> RUN python3 mwe.py
  56 |     
  57 |     RUN bpls -a -l test.bp
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 mwe.py" did not complete successfully: exit code: 1

[jorgensd]

This is now present on ubuntu:24.04 with numpy==2.1.1. Following is a minimal dockerfile (using the same code as above to verify the Python code):

FROM ubuntu:24.04

ARG MPI="openmpi"
ENV OPENBLAS_NUM_THREADS=1 \
    OPENBLAS_VERBOSE=0
ENV DEB_PYTHON_INSTALL_LAYOUT=deb_system
ENV DEBIAN_FRONTEND=noninteractive
ARG ADIOS2_VERSION=v2.10.0

WORKDIR /tmp

RUN apt-get -qq update && \
    apt-get -yq  upgrade && \
    apt-get -y install \
    lib${MPI}-dev \
    python3-dev \
    python3-pip \
    python3-setuptools \
    python3-venv \
    git \
    cmake \
    ninja-build \
    build-essential && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

ENV VIRTUAL_ENV=/test-env
ENV PATH=${VIRTUAL_ENV}/bin:$PATH
RUN python3 -m venv ${VIRTUAL_ENV}

# Install Python packages (via pip)
ARG NUMPY_VERSION=2.1.1
RUN python3 -m pip install --no-cache-dir mpi4py numpy==${NUMPY_VERSION}

ENV CMAKE_INSTALL_PYTHONDIR=${VIRTUAL_ENV}/lib/python3.12/site-packages

# BUILD ADIOS2
RUN git clone --branch=${ADIOS2_VERSION} --single-branch https://github.com/ornladios/ADIOS2.git adios2 && \
    cmake -G Ninja -DADIOS2_USE_HDF5=off -DADIOS2_USE_Python=on -DADIOS2_USE_Fortran=off -DBUILD_TESTING=off -DADIOS2_BUILD_EXAMPLES=off -DCMAKE_INSTALL_PYTHONDIR=${CMAKE_INSTALL_PYTHONDIR} -DADIOS2_USE_ZeroMQ=off -B build-dir-adios2 -S ./adios2 && \
    cmake -G Ninja -B build-dir-adios2 -DCMAKE_BUILD_TYPE="Release" -S ./adios2/ && \
    cmake --build build-dir-adios2 --parallel 2 && \
    cmake --install build-dir-adios2

ENV LD_LIBRARY_PATH=/usr/local/lib
WORKDIR /src/test

COPY mwe.py mwe.py
RUN python3 mwe.py -v
RUN bpls -a -l test.bp

Note that this has seems to have been fixed on the master branch of ADIOS2, so it would be good with a new release.

[pnorbert]

Thank you for the report. I can reproduce this on ubuntu 24.04 and adios2 release_210 branch. And as you state, the master branch works fine.

[finsberg]

I have the same issue. Would it be possible to create a new release of adios2?

[pnorbert]

We decided to make a 2.10.2 release with a handful of bug fixes, including this one. Stay tuned.

[jorgensd]

We decided to make a 2.10.2 release with a handful of bug fixes, including this one. Stay tuned.

Any update on this?

[eisenhauer]

Yes, it was delayed by vacation, but we're hoping to have 2.10.2 out by the end of the week.