Installation issues: cmake failing on FindNumpy

[ChristopherAJDuncan]

Hi there. New Mac (M3 chip) and doing a fresh install of everything. Unfortunately I'm falling over on getting pyccl to install. I've tried a few permutations, but with the same failures:

1. Using python3.12 installed via the official python dmg file (located in /Library/Frameworks/Python.frameworks)
2. Using python3.11 installed via macports (in /opt/local/bin) - this worked on a previous laptop

I've also tried in both cases installing in a venv. I've tried installing without numpy (and indeed it compiles numpy as part of the install), and with numpy explicitly installed.

The error message I get is as follows (following pip3 install pyccl):

Building wheels for collected packages: pyccl Building wheel for pyccl (pyproject.toml) ... error error: subprocess-exited-with-error

× Building wheel for pyccl (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [171 lines of output] WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section' running bdist_wheel running build running build_py -- No build type selected, default to Release -- CMAKE_BUILD_TYPE: Release -- The C compiler identification is AppleClang 15.0.0.15000100 -- The CXX compiler identification is AppleClang 15.0.0.15000100 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done CMake Warning (dev) at /opt/local/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:438 (message): The package name passed to find_package_handle_standard_args (PkgConfig) does not match the name of the calling package (FFTW). This can lead to problems in calling code that expects find_package result variables (e.g., _FOUND) to follow a certain pattern. Call Stack (most recent call first): /opt/local/share/cmake-3.24/Modules/FindPkgConfig.cmake:99 (find_package_handle_standard_args) cmake/Modules/FindFFTW.cmake:18 (include) cmake/Modules/BuildFFTW.cmake:6 (find_package) CMakeLists.txt:51 (include) This warning is for project developers. Use -Wno-dev to suppress it.

  -- Found PkgConfig: /opt/local/bin/pkg-config (found version "0.29.2")
  -- Could NOT find FFTW (missing: FFTW_INCLUDES FFTW_LIBRARIES)
  -- FFTW not found, downloading and compiling from source
  CMake Warning (dev) at /opt/local/share/cmake-3.24/Modules/ExternalProject.cmake:3074 (message):
    The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
    not set.  The policy's OLD behavior will be used.  When using a URL
    download, the timestamps of extracted files should preferably be that of
    the time of extraction, otherwise code that depends on the extracted
    contents might not be rebuilt if the URL changes.  The OLD behavior
    preserves the timestamps from the archive instead, but this is usually not
    what you want.  Update your project to the NEW behavior or specify the
    DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
    robustness issue.
  Call Stack (most recent call first):
    /opt/local/share/cmake-3.24/Modules/ExternalProject.cmake:4170 (_ep_add_download_command)
    cmake/Modules/BuildFFTW.cmake:11 (ExternalProject_Add)
    CMakeLists.txt:51 (include)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Found GSL: /opt/local/include (found suitable version "2.7.1", minimum required is "2.1")
  -- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
  -- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
  -- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
  -- Found SWIG: /opt/local/bin/swig (found version "4.1.1")
  -- Found PythonInterp: /Users/Shared/env/py3.12/bin/python3.12 (found suitable version "3.12.2", minimum required is "3.12.2")
  -- Found PythonLibs: /Library/Frameworks/Python.framework/Versions/3.12/lib/libpython3.12.dylib
  -- Found PythonInterp: /Users/Shared/env/py3.12/bin/python3.12 (found version "3.12.2")
  CMake Error at cmake/Modules/FindNumPy.cmake:70 (list):
    list GET given empty list
  Call Stack (most recent call first):
    pyccl/CMakeLists.txt:9 (find_package)

  CMake Error at cmake/Modules/FindNumPy.cmake:71 (list):
    list GET given empty list
  Call Stack (most recent call first):
    pyccl/CMakeLists.txt:9 (find_package)

  CMake Error at cmake/Modules/FindNumPy.cmake:74 (string):
    string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
    command.
  Call Stack (most recent call first):
    pyccl/CMakeLists.txt:9 (find_package)

  CMake Error at cmake/Modules/FindNumPy.cmake:77 (string):
    string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
    command.
  Call Stack (most recent call first):
    pyccl/CMakeLists.txt:9 (find_package)

  CMake Error at cmake/Modules/FindNumPy.cmake:81 (string):
    string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty
    string.
  Call Stack (most recent call first):
    pyccl/CMakeLists.txt:9 (find_package)

  CMake Error at cmake/Modules/FindNumPy.cmake:82 (math):
    math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) +
    NOTFOUND": syntax error, unexpected exp_TIMES (11).
  Call Stack (most recent call first):
    pyccl/CMakeLists.txt:9 (find_package)

  CMake Warning (dev) at /opt/local/share/cmake-3.24/Modules/UseSWIG.cmake:775 (message):
    Policy CMP0078 is not set: UseSWIG generates standard target names.  Run
    "cmake --help-policy CMP0078" for policy details.  Use the cmake_policy
    command to set the policy and suppress this warning.

  Call Stack (most recent call first):
    pyccl/CMakeLists.txt:28 (swig_add_library)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Warning (dev) at /opt/local/share/cmake-3.24/Modules/UseSWIG.cmake:617 (message):
    Policy CMP0086 is not set: UseSWIG honors SWIG_MODULE_NAME via -module
    flag.  Run "cmake --help-policy CMP0086" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.

  Call Stack (most recent call first):
    /opt/local/share/cmake-3.24/Modules/UseSWIG.cmake:888 (SWIG_ADD_SOURCE_TO_MODULE)
    pyccl/CMakeLists.txt:28 (swig_add_library)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Configuring incomplete, errors occurred!
  See also "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-install-i_3rhrtz/pyccl_6fbd45a12fdb4f0086f462260561c8a0/build/CMakeFiles/CMakeOutput.log".
  See also "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-install-i_3rhrtz/pyccl_6fbd45a12fdb4f0086f462260561c8a0/build/CMakeFiles/CMakeError.log".
  Traceback (most recent call last):
    File "/Users/Shared/env/py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/Shared/env/py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Shared/env/py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 404, in build_wheel
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 480, in run_setup
      super().run_setup(setup_script=setup_script)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 82, in <module>
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/normal/lib/python3.12/site-packages/wheel/bdist_wheel.py", line 368, in run
      self.run_command("build")
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/private/var/folders/rf/0gffwms124s2pyftslxm1nlc0000gp/T/pip-build-env-cpj4rg9h/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "<string>", line 56, in run
    File "<string>", line 22, in _compile_ccl
  Exception: Could not run CMake configuration. Make sure CMake is installed !
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pyccl Failed to build pyccl ERROR: Could not build wheels for pyccl, which is required to install pyproject.toml-based projects

I've also tried explicitly setting:

export CC=/opt/local/bin/arm64-apple-darwin23-gcc-13.2.0 export LD_LIBRARY_PATH=/opt/local/lib/ export LDFLAGS=-L/opt/local/lib/ export CPPFLAGS=-I/opt/local/include/ to tell the code where to find macport installs. Notably it still doesn't find FFTW, even though that is install by macports, so this might indicate some other linking error.

Any help greatly appreciated: I've exhausted my know-how unfortunately...

[ChristopherAJDuncan]

Under certain conditions (if I run setup.py explicitly) I get the following:

... Scanning dependencies of target ccllib_swig_compilation [ 93%] Swig compile ccl.i for python :3: Error: Unable to find 'python.swg ...

[tilmantroester]

The quick solution, unless you absolutely need to use the system or macports python, is to use conda.

[hsinfan1996]

Check if you can import numpy in your python interpreters. cmake obtains include path using python -c "import numpy as n; print(n.__version__); print(n.get_include());", with python replaced by the path to the python interpreter found previously.

[ChristopherAJDuncan]

Thanks @tilmantroester, I can install with Conda, but the script requires use of namaster (which is not currently hosted on Conda in arm form as far as I can tell, but also is another project I'm struggling to install anyway...) and glass (which is only hosted on PyPi). I'm not very familiar with Conda, and I think it can be set up to mix Conda forge with PyPi so I'm willing to try, but ideally could rely on PyPi entirely.

@hsinfan1996 thanks for the suggestion: the python interpreters can all see numpy. Even running your suggestion outputs:

1.26.3 /Users/Shared/env/py3.11/lib/python3.11/site-packages/numpy/core/include

as expected

[ChristopherAJDuncan]

Aha, I got it. I can install directly using python setup.py install. There were only two issues:

1. I had installed swig using macports, but had not pip installed swig
2. I had to force a completely clean install.

Thanks to all who responded. One step forward...

I'd note that I still cannot install via PyPi, but at least I have a working version now :)