Open VictorEijkhout opened 2 months ago
[staff corrfunc:100] find $TACC_PYTHON_DIR -name arrayobject.h
/work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/_core/include/numpy/arrayobject.h
I think that's a Numpy 2 path. Can you try with Numpy < 2?
In file included from /tmp/pip-build-env-2zutweuw/normal/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarrayobject.h:22,
from /tmp/pip-build-env-2zutweuw/normal/lib/python3.11/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
from _countpairs.c:18:
/tmp/pip-build-env-2zutweuw/normal/lib/python3.11/site-packages/numpy/_core/include/numpy/__multiarray_api.h:1475:10: fatal error: numpy/_public_dtype_api_table.h: No such file or directory
1475 | #include "numpy/_public_dtype_api_table.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
but
/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/_public_dtype_api_table.h
so I think I need to adjust an include path.
Here is the compile line:
gcc -DVERSION=\"2.5.2\" -DUSE_UNICODE -std=c99 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -O3 -ftree-vectorize -funroll-loops -fprefetch-loop-arrays --param simultaneous-prefetches=4 -fopenmp -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -I../../theory/DD -I../../theory/DDrppi -I../../theory/wp -I../../theory/xi -I../../theory/DDsmu -I../../theory/vpf -isystem/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/include/python3.11 -isystem /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/ -I../../io -I../../utils _countpairs.c -c -o _countpairs.o
In file included from /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/ndarrayobject.h:22,
from /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/arrayobject.h:5,
from _countpairs.c:18:
/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/__multiarray_api.h:1475:10: fatal error: numpy/_public_dtype_api_table.h: No such file or directory
1475 | #include "numpy/_public_dtype_api_table.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I don't understand who constructs this.
Hmm - this definitely looks like a numpy issue - similar to this one. May be this suggestion could solve the issue?
That should be #include <numpy/arrayobject.h>, not #include <arrayobject.h>.
Will you please confirm that that fix works?
@VictorEijkhout What version of numpy are you using?
Numpy 2.0.1
And that fix makes things worse:
make[3]: Entering directory '/work2/00434/eijkhout/corrfunc/corrfunc-2.5.2/theory/python_bindings'
gcc -DVERSION=\"2.5.2\" -DUSE_UNICODE -std=c99 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -O3 -ftree-vectorize -funroll-loops -fprefetch-loop-arrays --param simultaneous-prefetches=4 -fopenmp -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -I../../theory/DD -I../../theory/DDrppi -I../../theory/wp -I../../theory/xi -I../../theory/DDsmu -I../../theory/vpf -isystem/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/include/python3.11 -isystem /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/lib/python3.11/site-packages/numpy/_core/include/numpy/ -I../../io -I../../utils _countpairs.c -c -o _countpairs.o
_countpairs.c:18:10: fatal error: numpy/arrayobject.h: No such file or directory
18 | #include <numpy/arrayobject.h>
| ^~~~~~~~~~~~~~~~~~~~~
I think I need to add an include path, but I can't find where you set those.
My fix:
export NUMPY_INCL_FLAG := $(shell $(PYTHON) -c "from __future__ import print_function; im\port numpy; print('-isystem ' + numpy.__path__[0] + '/_core/include/numpy/' )" )
export NUMPY_INCL_FLAG2 := $(shell $(PYTHON) -c "from __future__ import print_function; i\mport numpy; print('-isystem ' + numpy.__path__[0] + '/_core/include/' )" )
[ ... stuff .... ]
endif
export NUMPY_CHECKED:=1
endif
export PYTHON_CFLAGS := $(PYTHON_INCL) $(NUMPY_INCL_FLAG) $(NUMPY_INCL_FLAG2)
Corrfunc hasn't yet been updated to support NumPy 2, but I took a quick stab at it and it seems to work (it builds and passes tests). I put it here if you'd like to try it: https://github.com/manodeep/Corrfunc/tree/numpy-2
It breaks NumPy 1 build support, which I'm not sure we want to do on main yet, so I'm going to leave it in a separate branch for now.
Note that in that branch NumPy 2 is just a build dependency, and it should run with a recent NumPy 1 as well.
Numpy 2.0.1
And that fix makes things worse:
make[3]: Entering directory '/work2/00434/eijkhout/corrfunc/corrfunc-2.5.2/theory/python_bindings' gcc -DVERSION=\"2.5.2\" -DUSE_UNICODE -std=c99 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -O3 -ftree-vectorize -funroll-loops -fprefetch-loop-arrays --param simultaneous-prefetches=4 -fopenmp -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -I../../theory/DD -I../../theory/DDrppi -I../../theory/wp -I../../theory/xi -I../../theory/DDsmu -I../../theory/vpf -isystem/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/include/python3.11 -isystem /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/lib/python3.11/site-packages/numpy/_core/include/numpy/ -I../../io -I../../utils _countpairs.c -c -o _countpairs.o _countpairs.c:18:10: fatal error: numpy/arrayobject.h: No such file or directory 18 | #include <numpy/arrayobject.h> | ^~~~~~~~~~~~~~~~~~~~~
I think I need to add an include path, but I can't find where you set those.
Ahh never mind then. Looks like Lehman has a potential fix for numpy2
@lgarrison I made a suggestion on the commit - will you be able to check whether my suggestion works for both numpy1 and numpy2?
@VictorEijkhout I merged in the numpy2 capability and made a new release. Do you mind checking whether the code now compiles with numpy2 (either from source or from the PyPI repo with Corrfunc v2.5.3)?
I get an error on numpy headers not found, so I edit line 518 in common.mk to
which outputs
so I go searching for that directory:
Are you sure that path should exist?