search

manodeep / Corrfunc

⚡️⚡️⚡️Blazing fast correlation functions on the CPU.
https://corrfunc.readthedocs.io
MIT License
163 stars 50 forks source link

M2 compiling issues #294

Closed quasars100 closed 1 year ago

quasars100 commented 1 year ago

General information

Issue description

pip: ERROR: Failed building wheel for Corrfunc Running setup.py clean for Corrfunc Failed to build Corrfunc ERROR: Could not build wheels for Corrfunc, which is required to install pyproject.toml-based projects

GitHub directory results in the following error after "make install": clang: error: the clang compiler does not support '-march=native'

What have you tried so far?

Changing the "-march=native" lines in common.mk to "-mpu-apple=a14", adding make CC=gcc, updating my gsl/gcc/python/pip packages - still I get the same error.

Minimal failing example

** could not install

make install

# issue

Makefile:9: Not compiling C extensions for mocks. Either python or numpy not available gcc -DVERSION=\"2.5.0\" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3 -funroll-loops -march=native -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs logbins.c -lm -o logbins clang: error: the clang compiler does not support '-march=native' make[1]: [logbins] Error 1 make: [install] Error 2

manodeep commented 1 year ago

Thanks for using Corrfunc and for reporting the issue. I have been meaning to add the compilation-on-m1/m2 fix from the arm64 branch into the master branch and this gives me the reminder to do so now.

Even after the compilation is fixed, you still will have another issue with the non-detection of python/numpy. Can you add a bit more detail about your python/numpy setup - are you using conda/macports/brew?

manodeep commented 1 year ago

@quasars100 Will you please do the following to check if the new commits fix your compilation issue? The steps would be:

  1. git clone <repo>
  2. git checkout fix-compile-on-apple-soc
  3. make clean
  4. make

This should compile all of the command-line executables. If you encounter any warning/errors, please report back.

edit Fixed step 2: git checkout -b fix-compile-on-apple-soc git checkout fix-compile-on-apple-soc

quasars100 commented 1 year ago

Even after the compilation is fixed, you still will have another issue with the non-detection of python/numpy. Can you add a bit more detail about your python/numpy setup - are you using conda/macports/brew?

Thank you so much for the quick reply. My python version is Python 3.10.12 (installed using brew) and NumPy version is 1.24.3 (installed using pip). Local environment (no virtualenv), terminal, packages installed using brew (no condo/macports).

quasars100 commented 1 year ago

@quasars100 Will you please do the following to check if the new commits fix your compilation issue? The steps would be:

  1. git clone <repo>
  2. git checkout -b fix-compile-on-apple-soc
  3. make clean
  4. make

This should compile all of the command-line executables. If you encounter any warning/errors, please report back.

I tried running the code block above and unfortunately got the following error:

In file included from ../../utils/cpu_features.c:13: ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) ); ^ ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' ../../utils/cpu_features.h:69:10: error: couldn't allocate output register for constraint 'a' asm("xgetbv" : "=a"(a),"=d"(d) : "c"(ctr) : ); ^ ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' __asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) ); ^ ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' 5 errors generated. make[2]: [../../utils/cpu_features.o] Error 1 make[1]: [DD] Error 2 make: *** [theory] Error 2

manodeep commented 1 year ago

That is odd - those lines of code should never be seen by the compiler if the environment variable __ARM_NEON is defined - which is supposed to be the case. For reference, I just did a fresh repo clone and tried out my instructions and those worked. I also notice that there is line number mis-match between what I see and what you have reported in the error - is it possible that you accidentally skipped the checkout branch line? There was an error in what I quoted - you should do git checkout fix-compile-on-apple-soc (i.e., without the -b )

If you were indeed in the correct branch, will you please report back on the following steps:

clang -DOUTPUT_RPAVG   -DENABLE_MIN_SEP_OPT  -DCOPY_PARTICLES  -DLINK_IN_DEC -DLINK_IN_RA  -DDOUBLE_PREC -DUSE_OMP  -DVERSION=\"2.5.0\" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3  -fopenmp=libomp -funroll-loops -fno-strict-aliasing -Wformat=2  -Wpacked  -Wnested-externs -Wpointer-arith  -Wredundant-decls  -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes  -Wnested-externs -Wstrict-prototypes   -Wno-unused-local-typedefs  -I../../io -I../../utils -I/opt/local/include -I../../mocks/DDrppi_mocks -I../../mocks/DDsmu_mocks -I../../mocks/DDtheta_mocks -I../../mocks/vpf_mocks -c run_correlations_mocks.c -o run_correlations_mocks.o

And then, can you please report back on these two things at the command-line:

quasars100 commented 1 year ago

Here is what the output is after the following steps. In terminal, in the ~/Corrfunc directory:

(1. git clone ) --> already cloned and in the Corrfunc directory

  1. git checkout fix-compile-on-apple-soc
  2. make clean
  3. make

Output (in terminal):

In file included from ../../utils/cpu_features.c:13: ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) ); ^ ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' ../../utils/cpu_features.h:69:10: error: couldn't allocate output register for constraint 'a' asm("xgetbv" : "=a"(a),"=d"(d) : "c"(ctr) : ); ^ ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' __asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) ); ^ ../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a' 5 errors generated. make[2]: [../../utils/cpu_features.o] Error 1 make[1]: [DD] Error 2 make: *** [theory] Error 2 achen@MAC0008879 Corrfunc %

Unfortunately, still an error appears. Also, thank you for all your help so far, and the compilers are listed below

quasars100 commented 1 year ago

And then, can you please report back on these two things at the command-line:

  • which <compiler> (where you replace the with whichever compiler that you are using).

which clang clang: aliased to /opt/homebrew/Cellar/llvm/16.0.5/bin/clang

which gcc gcc: aliased to /opt/homebrew/Cellar/gcc/13.1.0/bin/gcc-13

  • <compiler> -mcpu=apple-m1 -dM -E - < /dev/null | grep -i ARM

achen@MAC0008879 ~ % clang -mcpu=apple-m1 -dM -E - </dev/null | grep -i ARM` pipe bquote>

manodeep commented 1 year ago

Just to confirm: there are no matches to the ARM grep command for either of the compilers?

manodeep commented 1 year ago

Also, what happens if you do make CC=gcc (after a make clean)

What’s happening is that the compiler is not supposed to see that chunk of code on ARM, based on a variable usually defined by the compiler. That assumption of defined variable seems to break down on your setup. Since I don’t use brew (macports person) here, I hadn’t encountered this issue before

lgarrison commented 1 year ago

Since @quasars100 probably ran the original instructions with git checkout -b fix-compile-on-apple-soc, they probably have a local branch that's not tracking the remote. I think the steps to fix this are:

$ git checkout master
$ git branch -d fix-compile-on-apple-soc
$ git checkout fix-compile-on-apple-soc

I think at that point, you'll have a local branch that's tracking the remote, which you can confirm with git branch -vv or git status. Then hopefully the code will compile.

quasars100 commented 1 year ago

Just to confirm: there are no matches to the ARM grep command for either of the compilers?

There aren't any matches returning to the ARM grep command for either clang or gcc compilers:

achen@MAC0008879 ~ % which clang clang: aliased to /opt/homebrew/Cellar/llvm/16.0.5/bin/clang achen@MAC0008879 ~ % which gcc gcc: aliased to /opt/homebrew/Cellar/gcc/13.1.0/bin/gcc-13 achen@MAC0008879 ~ % clang -mcpu=apple-m1 -dM -E - </dev/null | grep -i ARM pipe bquote> pipe bquote> achen@MAC0008879 ~ % gcc -mcpu=apple-m1 -dM -E - </dev/null | grep -i ARM

pipe bquote> pipe bquote>

quasars100 commented 1 year ago

Also, what happens if you do make CC=gcc (after a make clean)

What’s happening is that the compiler is not supposed to see that chunk of code on ARM, based on a variable usually defined by the compiler. That assumption of defined variable seems to break down on your setup. Since I don’t use brew (macports person) here, I hadn’t encountered this issue before

This seems to be able to run, since the last few lines of output are:

ar rcs libcountspheres_mocks.a countspheres_mocks.o countspheres_mocks_impl_float.o countspheres_mocks_impl_double.o ../../utils/gridlink_impl_float.o ../../utils/gridlink_impl_double.o ../../utils/gridlink_utils_float.o ../../utils/gridlink_utils_double.o ../../utils/utils.o ../../utils/progressbar.o ../../utils/cpu_features.o ../../utils/avx512_calls.o ../../utils/set_cosmo_dist.o ../../utils/cosmology_params.o make[2]: Nothing to be done for `lib'. /Library/Developer/CommandLineTools/usr/bin/make -C examples gcc -DOUTPUT_RPAVG -DENABLE_MIN_SEP_OPT -DCOPY_PARTICLES -DLINK_IN_DEC -DLINK_IN_RA -DDOUBLE_PREC -DUSE_OMP -DVERSION=\"2.5.0\" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3 -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -I../../io -I../../utils -I/opt/homebrew/Cellar/gsl/2.7.1/include -I../../mocks/DDrppi_mocks -I../../mocks/DDsmu_mocks -I../../mocks/DDtheta_mocks -I../../mocks/vpf_mocks -c run_correlations_mocks.c -o run_correlations_mocks.o gcc run_correlations_mocks.o ../../io/ftread.o ../../io/io.o ../../utils/utils.o ../../utils/progressbar.o ../../utils/cosmology_params.o ../../mocks/DDrppi_mocks/libcountpairs_rp_pi_mocks.a ../../mocks/DDsmu_mocks/libcountpairs_s_mu_mocks.a ../../mocks/DDtheta_mocks/libcountpairs_theta_mocks.a ../../mocks/vpf_mocks/libcountspheres_mocks.a -lm -L/opt/homebrew/Cellar/gsl/2.7.1/lib -lgsl -lgslcblas -Xlinker -rpath -Xlinker /opt/homebrew/Cellar/gsl/2.7.1/lib -o run_correlations_mocks achen@MAC0008879 Corrfunc %

However, in the middle, there was the following warning:

If you want to permanently set the default compiler to gcc for all future compilations, please update the "CC" variable in "common.mk" ../common.mk:276: Compiler is Apple clang and does not support OpenMP If you want OpenMP support, please install clang with OpenMP support For homebrew, use "brew update && (brew outdated xctool || brew upgrade xctool) && brew tap homebrew/versions && brew install clang-omp" For Macports, use "sudo port install clang-3.8 +assertions +debug + openmp" ../common.mk:320: $CC = gcc does not support OpenMP - please use gcc/icc for compiling with openmp. Removing "USE_OMP" from compile options. If you are sure your version of "clang" ( must be >= 3.7, NOT Apple clang) does support OpenMP, then comment out the line "OPT:=$(filter-out -DUSE_OMP,$(OPT))" in the file "common.mk" You might have to add in the include path (path to "omp.h") to "CFLAGS" and the runtime library path to "CLINK" at the top of "common.mk" /bin/sh: python: command not found /bin/sh: python: command not found /bin/sh: line 0: [: too many arguments /bin/sh: line 0: [: too many arguments ../common.mk:497: Found python version .. but minimum required python is 2.7 ../common.mk:502: Found NUMPY version .. but minimum required numpy is 1.7

-------COMPILE SETTINGS------------ MAKE = ["/Library/Developer/CommandLineTools/usr/bin/make"] CC = ["gcc"] OPT = ["-DOUTPUT_RPAVG -DENABLE_MIN_SEP_OPT -DCOPY_PARTICLES -DLINK_IN_DEC -DLINK_IN_RA -DDOUBLE_PREC"] CFLAGS = ["-DVERSION=\"2.5.0\" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3 -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -mcpu=apple-m1 -mtune=apple-m1"] CLINK = [" -lm"] PYTHON = ["python"] GSL_CFLAGS = [""] GSL_LINK = [""] PYTHON_CFLAGS = [""] -------END OF COMPILE SETTINGS------------

Makefile:7: Not compiling C extensions for mocks. Either python or numpy not available

quasars100 commented 1 year ago

However, when I check my python version, it was 3.10, which is > python2.7

achen@MAC0008879 Corrfunc % which python python: aliased to /opt/homebrew/bin/python3.10

And numpy:

Python 3.10.12 (main, Jun 7 2023, 00:37:42) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import numpy numpy.version '1.24.3'

which is >numpy 1.7.

Sorry for how long this issue seems to be, and thank you all again for all your help.

quasars100 commented 1 year ago

Since @quasars100 probably ran the original instructions with git checkout -b fix-compile-on-apple-soc, they probably have a local branch that's not tracking the remote. I think the steps to fix this are:

$ git checkout master
$ git branch -d fix-compile-on-apple-soc
$ git checkout fix-compile-on-apple-soc

I think at that point, you'll have a local branch that's tracking the remote, which you can confirm with git branch -vv or git status. Then hopefully the code will compile.

Thank you, and after running these steps, the code did seem to compile, however, this was the end output:

Makefile:7: Not compiling C extensions for mocks. Either python or numpy not available /Library/Developer/CommandLineTools/usr/bin/make -C DDrppi_mocks make[2]: Nothing to be done for all'. /Library/Developer/CommandLineTools/usr/bin/make -C DDtheta_mocks make[2]: Nothing to be done forall'. /Library/Developer/CommandLineTools/usr/bin/make -C DDsmu_mocks make[2]: Nothing to be done for all'. /Library/Developer/CommandLineTools/usr/bin/make -C vpf_mocks make[2]: Nothing to be done forall'. for dir in DDrppi_mocks DDtheta_mocks DDsmu_mocks vpf_mocks examples; do \ /Library/Developer/CommandLineTools/usr/bin/make -C $dir lib; \ done make[2]: Nothing to be done for lib'. make[2]: Nothing to be done forlib'. make[2]: Nothing to be done for lib'. make[2]: Nothing to be done forlib'. make[2]: Nothing to be done for lib'. /Library/Developer/CommandLineTools/usr/bin/make -C examples make[2]: Nothing to be done forall'. achen@MAC0008879 Corrfunc %

Also, when I try calling "Corrfunc" in python and using its commands, I get the following error:

pairs = xi(L, 2, bins, pos[:,0], pos[:,1], pos[:,2]) Traceback (most recent call last): File "/Users/achen/Corrfunc/Corrfunc/theory/xi.py", line 187, in xi from Corrfunc._countpairs import countpairs_xi as xi_extn ModuleNotFoundError: No module named 'Corrfunc._countpairs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/Users/achen/Corrfunc/Corrfunc/theory/xi.py", line 191, in xi raise ImportError(msg) ImportError: Could not import the C extension for the correlation function.

The package does import in python without issue, it's just when I try running the commands they don't work.
Thanks again, and sorry for the myriad of issues this seems to have on brew.

lgarrison commented 1 year ago

Thanks, that seems like progress! @manodeep, do you know why the Python detection would fail with brew Python?

manodeep commented 1 year ago

Hoorrray - that's definitely progress! At least the code seems to be compiling (which I am surprised by, given the lack of the (usually) defined ARM variables but I will put that down to a bad grep implementation). @quasars100 Please don't apologize - you are helping us improve Corrfunc, and that, in turn, helps everyone - so thank you!

@lgarrison Not really sure why python is not being detected - the detection is simply a shell calling python! I see that the output says /bin/sh: python: command not found. My guess is that python is defined as an alias and, therefore, only available on the interactive terminal (i.e., where @quasars100 is working but not through the non-interactive terminal invoked by make). @quasars100 What do you get if you try alias python? And regardless of the alias status, using python -m pip install -e . --verbose should work. Even if python is aliased, that specific command should use the correct python.

Related, I see that OpenMP is not enabled; ideally, that also needs to be fixed. @quasars100 Will you please run brew install llvm to install the OpenMP-enabled version of clang and then recompile (after make clean) by invoking make (or equivalently, make CC=clang)

manodeep commented 1 year ago

@quasars100 To explicitly clarify, as long as python and numpy is not detected at compile time, the Corrfunc C extensions will not be compiled, and therefore, you will not be able to use Corrfunc from python

quasars100 commented 1 year ago

Hoorrray - that's definitely progress! At least the code seems to be compiling (which I am surprised by, given the lack of the (usually) defined ARM variables but I will put that down to a bad grep implementation). @quasars100 Please don't apologize - you are helping us improve Corrfunc, and that, in turn, helps everyone - so thank you!

@lgarrison Not really sure why python is not being detected - the detection is simply a shell calling python! I see that the output says /bin/sh: python: command not found. My guess is that python is defined as an alias and, therefore, only available on the interactive terminal (i.e., where @quasars100 is working but not through the non-interactive terminal invoked by make). @quasars100 What do you get if you try alias python? And regardless of the alias status, using python -m pip install -e . --verbose should work. Even if python is aliased, that specific command should use the correct python.

Here is what I get when running "alias python" in terminal:

alias python python=/opt/homebrew/bin/python3.10

(which python python: aliased to /opt/homebrew/bin/python3.10 )

Related, I see that OpenMP is not enabled; ideally, that also needs to be fixed. @quasars100 Will you please run brew install llvm to install the OpenMP-enabled version of clang and then recompile (after make clean) by invoking make (or equivalently, make CC=clang)

So prior to opening this issue, I actually ran "brew install llvm", and that is the version of clang my current clang compiler is aliased to, which is why the warning of "OpenMP is not available with apple clang" was confusing, since clang should be aliased away from Apple clang now.

After running "make CC=clang", OpenMP is enabled, but I still get this error:

/bin/sh: python: command not found /bin/sh: python: command not found /bin/sh: line 0: [: too many arguments /bin/sh: line 0: [: too many arguments ../common.mk:497: Found python version .. but minimum required python is 2.7 ../common.mk:502: Found NUMPY version .. but minimum required numpy is 1.7

However when I check the python version:

which python python: aliased to /opt/homebrew/bin/python3.10 achen@MAC0008879 Corrfunc % python Python 3.10.12 (main, Jun 7 2023, 00:37:42) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import numpy numpy.version '1.24.3'

Python > 2.7 and numpy > 1.7. Do you know if this is something to be changed in "common.mk" or my local environment?

Thank you again so much for all your help!

manodeep commented 1 year ago

Ohh good - so python is defined as an alias. No wonder make can't find python.

Try and run export CC=clang && python -m pip install -e . - that should do the trick

quasars100 commented 1 year ago

Yes, that worked! (Has to be in the Corrfunc cloned directory though). Thank you so much!

manodeep commented 1 year ago

Great - glad this all worked :)! Can you confirm that you have OpenMP support, i.e., that you can use multiple threads now?

Sidenote: The reason why you were facing the compilation and non-detection of llvm-clang/python/numpy is all because you (or homebrew) have the relevant aliases set. The aliases work fine for you since you are always accessing from an interactive terminal - i.e., a terminal where you can send keyboard input to. The shell sessions spawned by common.mk are non-interactive and, therefore, do not know about the aliases.

I also suspect your PATH variable does not contain /opt/homebrew/bin (for llvm-clang) ahead of /usr/bin (for apple-clang), and thus, when the non-interactive shell resolves clang as /usr/bin/clang. Since I don't use homebrew, I am unsure if it is an intentional design that /opt/homebrew/bin is not added to your PATH environment variable. If you ever run into such odd issues of not-detecting-the-correct-tool, then it might be worthwhile adding the full path to whichever tool that you are wanting to detect (or equivalently, add /opt/homebrew/bin to PATH either in the terminal you are working in, or in the startup shell file - .bashrc/.zshrc etc)

quasars100 commented 1 year ago

Thank you! Yes, OpenMP works now, and the tests (verify installation - https://corrfunc.readthedocs.io/en/master/install.html) have passed. I will try exporting the PATH variable in my .zsh file and see if that works for future installations.