search

ornladios / ADIOS

The old ADIOS 1.x code repository. Look for ADIOS2 for new repo
https://csmd.ornl.gov/adios
Other
54 stars 40 forks source link

Python 3.9 Build Error #216

Open ax3l opened 3 years ago

ax3l commented 3 years ago

Building against CPython 3.9 I see the following errors:

/home/conda/feedstock_root/build_artifacts/adios-python_1602593380049/_build_env/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
adios.cpp: In function 'PyObject* PyInit_adios()':
adios.cpp:52250:30: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
   __pyx_type_5adios_softdict.tp_print = 0;
                              ^~~~~~~~
                              tp_dict

Refs.: https://github.com/conda-forge/adios-python-feedstock

cc @jychoi-hpc

ax3l commented 2 years ago

Ping @jychoi-hpc @pnorbert @dmitry-ganyushin @lwan86: do you think this can be easily fixed? :)

jychoi-hpc commented 2 years ago

@ax3l sorry for my late reply. Can you help me to understand the error? Is this because of using new version of numpy? which python version did you use? I tried to build on Cori with python 3.9.7 (numpy 1.20.3, cython 0.29.24) and didn't get an error.

ax3l commented 2 years ago

@jychoi-hpc no worries. I think this is due to a new version of CPython. tp_print is dropped from Python 3.9 onwards. https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset

Maybe Cython of some of the checked-in ADIOS Cython calls/file generations need to be updated?

ax3l commented 2 years ago

I see the problem on conda-forge: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset and build like this: https://github.com/conda-forge/adios-python-feedstock/blob/master/recipe/build.sh

Should I run Cython or something beforehand? :)

jychoi-hpc commented 2 years ago

I just tried again by doing exactly the same thing in build.sh on Cori without any issue. I used Python 3.9.7. It looks like condo-forge is using more recent python. Do you know which version of python on conda-forge is used?

Alternatively, you can try the following command to invoke 'cython' beforehand:

make CYTHON=y python

instead of 

make python
ax3l commented 2 years ago

Thanks a lot, let's try this in Spack (https://github.com/spack/spack/pull/28003 / https://github.com/spack/spack/pull/28223) and Conda-Forge (https://github.com/conda-forge/adios-python-feedstock/pull/11).

X-ref: #202

ax3l commented 2 years ago

Works, fantastic! Thanks for the hint!

I noticed that the makefile does not reliably re-create the .cpp files from the .pyx files. In Spack, I had to remove them beforehand and disable make parallelism (-j) to get it to build: https://github.com/spack/spack/pull/28223

Yikun commented 1 year ago

Looks like we still hit this error with suggested commands:

[  214s] + make CYTHON=y MPI=y python
[  214s] python setup.py build_ext -lrt
[  214s] running build_ext
[  214s] building 'adios.adios' extension
[  214s] creating build
[  214s] creating build/temp.linux-aarch64-3.9
[  214s] gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -D_GNU_SOURCE -fPIC -fwrapv -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/home/abuild/rpmbuild/BUILD/adios-1.13.1/src/public -L/home/abuild/rpmbuild/BUILD/adios-1.13.1/src -I/opt/ohpc/pub/mpi/openmpi4-gnu12/4.1.4/include -fPIC -D_NOMPI -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/adios/1.13.1/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/netcdf/4.9.0/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/hdf5/1.10.8/include -I/usr/include -I/usr/include/python3.9 -c adios.cpp -o build/temp.linux-aarch64-3.9/adios.o -Wno-uninitialized -Wno-unused-function
[  214s] In file included from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
[  214s]                  from adios.cpp:540:
[  214s] /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
[  214s]    17 | #warning "Using deprecated NumPy API, disable it with " \
[  214s]       |  ^~~~~~~
[  215s] adios.cpp: In function ‘PyObject* PyInit_adios()’:
[  215s] adios.cpp:52250:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52250 |   __pyx_type_5adios_softdict.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52261:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52261 |   __pyx_type_5adios_file.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52287:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52287 |   __pyx_type_5adios_blockinfo.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52300:25: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52300 |   __pyx_type_5adios_var.tp_print = 0;
[  215s]       |                         ^~~~~~~~
[  215s] adios.cpp:52306:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52306 |   __pyx_type_5adios_attr.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52312:27: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52312 |   __pyx_type_5adios_group.tp_print = 0;
[  215s]       |                           ^~~~~~~~
[  215s] adios.cpp:52338:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52338 |   __pyx_type_5adios_smartdict.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52343:28: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52343 |   __pyx_type_5adios_writer.tp_print = 0;
[  215s]       |                            ^~~~~~~~
[  215s] adios.cpp:52348:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52348 |   __pyx_type_5adios_attrinfo.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52353:29: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52353 |   __pyx_type_5adios_varinfo.tp_print = 0;
[  215s]       |                             ^~~~~~~~

Any suggestion?

[1] https://build.openeuler.org/package/live_build_log/home:huangtianhua:ohpc/adios-gnu12-openmpi4/standard_aarch64/aarch64

martin-g commented 1 year ago

Trying to build ADIOS 1.13.1 (latest release, from Apr 18 2018) fails with the above problem. adios.cpp and adios_mpi.cpp are regenerated with Cython 0.29 at Oct 2 2019 with https://github.com/ornladios/ADIOS/commit/327e15919ce7369d556f6bb0d3b79af9066aed39 I think it is time for a new release!

jychoi-hpc commented 1 year ago

It looks like the cython version 0.29 at Oct 2 2019 is old. I just tried adios 1.13.1, cython 0.29.32, and python 3.9.13 and it works for me:

jyc@login4 numpy ((v1.13.1)) $ cython --version
Cython version 0.29.32
jyc@login4 numpy ((v1.13.1)) $ python --version
Python 3.9.13
jyc@login4 numpy ((v1.13.1)) $ make CYTHON=y MPI=y python
cython --cplus adios_mpi.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios_mpi.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
cython --cplus adios.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
python setup.py build_ext -lrt
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running build_ext
building 'adios.adios' extension
gcc -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -fPIC -D_NOMPI -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios.cpp -o build/temp.linux-ppc64le-cpython-39/adios.o -Wno-uninitialized -Wno-unused-function
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from adios.cpp:761:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
g++ -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios_nompi -lrt -lm -lrt -o build/lib.linux-ppc64le-cpython-39/adios/adios.cpython-39-powerpc64le-linux-gnu.so
python setup_mpi.py build
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running build
running build_src
running build_py
copying src_mpi/__init__.py -> build/lib.linux-ppc64le-cpython-39/adios_mpi
running build_ext
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler:    /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc
MPI C++ compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx
MPI F77 compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif77
MPI F90 compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif90
building 'adios_mpi.adios_mpi' extension
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios_mpi.cpp -o build/temp.linux-ppc64le-cpython-39/adios_mpi.o -Wno-unused-function -Wno-uninitialized
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from adios_mpi.cpp:762:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios_mpi.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios -lrt -lm -o build/lib.linux-ppc64le-cpython-39/adios_mpi/adios_mpi.cpython-39-powerpc64le-linux-gnu.so

(removed some warnings)

I don't know which version of cython starts supproting python 3.9. But, can you try with different cython versions?

martin-g commented 1 year ago

I wonder how this works for you with v1.13.1! https://raw.githubusercontent.com/ornladios/ADIOS/v1.13.1/wrappers/numpy/adios.cpp does not use checks like #if PY_VERSION_HEX < 0x030800B1 before trying to access .tp_print as it does in master

I'll try to investigate some more tomorrow!

ax3l commented 1 year ago

@martin-g @Yikun you are probably aware (and don't let me break our enthusiasm on getting old builds to work), but ADIOS1 is now deprecated and ADIOS2 is the new hot project: https://github.com/ornladios/ADIOS2

jychoi-hpc commented 1 year ago

I used the following command to re-generate adios.cpp and adios_mpi.cpp:

make CYTHON=y MPI=y python

As @ax3l said, I also recommend to use Adios2 if possible.

martin-g commented 1 year ago

Thank you for your help, @jychoi-hpc and @ax3l !

I didn't know it is possible to re-generate these files on the fly! This is what we did at https://github.com/openhpc/ohpc/pull/1635/files If we face other issues most probably we will move to ADIOS 2!

williamfgc commented 1 year ago

If we face other issues most probably we will move to ADIOS 2!

ADIOS2 also has more features that have been maintained for a few years.