search

Qiskit / qiskit-aer

Aer is a high performance simulator for quantum circuits that includes noise models
https://qiskit.github.io/qiskit-aer/
Apache License 2.0
481 stars 354 forks source link

Findpackage numpy fails when building locally #910

Closed mtreinish closed 3 years ago

mtreinish commented 4 years ago

Informations

What is the current behavior?

Running python setup.py install on my local system in a clean python environment fails trying to find the numpy package


--------------------------------------------------------------------------------
-- Trying "Ninja" generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
-- The C compiler identification is GNU 10.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 10.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/mtreinish/git/qiskit/qiskit-aer/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying "Ninja" generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /home/mtreinish/git/qiskit/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build
  Command:
    /home/mtreinish/git/qiskit/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake /home/mtreinish/git/qiskit/qiskit-aer -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/home/mtreinish/git/qiskit/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/home/mtreinish/git/qiskit/qiskit-aer/test-dist-err/bin/python -DPYTHON_VERSION_STRING:STRING=3.8.5 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:FILEPATH=/usr/lib/libpython3.8.so -DSKBUILD:BOOL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/mtreinish/git/qiskit/qiskit-aer/test-dist-err/lib/python3.8/site-packages/skbuild/resources/cmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 -DCMAKE_BUILD_TYPE:STRING=Release

-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Conan: Automatic detection of conan settings from cmake
-- Conan: Settings= -s;build_type=Release;-s;compiler=gcc;-s;compiler.version=10;-s;compiler.libcxx=libstdc++11
-- Conan: checking conan executable
-- Conan: Found program /usr/bin/conan
-- Conan: Version found Conan version 1.25.2

-- Conan executing: /usr/bin/conan install . -s build_type=Release -s compiler=gcc -s compiler.version=10 -s compiler.libcxx=libstdc++11 -e=CONAN_CMAKE_PROGRAM=/home/mtreinish/git/qiskit/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake -g=cmake --build=missing
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
CONAN_CMAKE_PROGRAM=/home/mtreinish/git/qiskit/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake
conanfile.txt: Installing package
Requirements
    fmt/6.2.0 from 'conan-center' - Cache
    muparserx/4.0.8 from 'conan-center' - Cache
    nlohmann_json/3.1.1 from 'conan-center' - Cache
    spdlog/1.5.0 from 'conan-center' - Cache
Packages
    fmt/6.2.0:b173bbda18164d49a449ffadc1c9e817f49e819d - Cache
    muparserx/4.0.8:b173bbda18164d49a449ffadc1c9e817f49e819d - Cache
    nlohmann_json/3.1.1:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    spdlog/1.5.0:5ec35c7aba5410ef8b762c324eb39c457e8e6fe3 - Cache

Installing (downloading, building) binaries...
fmt/6.2.0: Already installed!
muparserx/4.0.8: Already installed!
nlohmann_json/3.1.1: Already installed!
spdlog/1.5.0: Already installed!
conanfile.txt: Generator cmake created conanbuildinfo.cmake
conanfile.txt: Generator txt created conanbuildinfo.txt
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo
-- Conan: Loading conanbuildinfo.cmake
-- Conan: Adjusting output directories
-- Conan: Using cmake targets configuration
-- Library spdlog found /home/mtreinish/.conan/data/spdlog/1.5.0/_/_/package/5ec35c7aba5410ef8b762c324eb39c457e8e6fe3/lib/libspdlog.a
-- Library muparserx found /home/mtreinish/.conan/data/muparserx/4.0.8/_/_/package/b173bbda18164d49a449ffadc1c9e817f49e819d/lib/libmuparserx.a
-- Library fmt found /home/mtreinish/.conan/data/fmt/6.2.0/_/_/package/b173bbda18164d49a449ffadc1c9e817f49e819d/lib/libfmt.a
-- Conan: Adjusting language standard
-- Current conanbuildinfo.cmake directory: /home/mtreinish/git/qiskit/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build
-- Conan: Compiler GCC>=5, checking major version 10
-- Conan: Checking correct version: 10
-- Looking for OpenMP support...
-- OpenMP found!
-- OpenMP_CXX_FLAGS = -fopenmp
-- OpenMP_EXE_LINKER_FLAGS = 
-- Looking for OpenBLAS library...
-- BLAS library found: /usr/lib/libopenblas.so
-- No Thrust support enabled
CMake Error at .eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find NumPy (missing: NumPy_INCLUDE_DIR) (found version "")
Call Stack (most recent call first):
  .eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  test-dist-err/lib/python3.8/site-packages/skbuild/resources/cmake/FindNumPy.cmake:98 (find_package_handle_standard_args)
  cmake/cython_utils.cmake:4 (find_package)
  qiskit/providers/aer/pulse/qutip_extra_lite/cy/CMakeLists.txt:2 (include)

-- Configuring incomplete, errors occurred!
See also "/home/mtreinish/git/qiskit/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build/CMakeFiles/CMakeOutput.log".
See also "/home/mtreinish/git/qiskit/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "/home/mtreinish/git/qiskit/qiskit-aer/test-dist-err/lib/python3.8/site-packages/skbuild/setuptools_wrap.py", line 582, in setup
    env = cmkr.configure(cmake_args,
  File "/home/mtreinish/git/qiskit/qiskit-aer/test-dist-err/lib/python3.8/site-packages/skbuild/cmaker.py", line 229, in configure
    raise SKBuildError(

An error occurred while configuring with CMake.
  Command:
    /home/mtreinish/git/qiskit/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake /home/mtreinish/git/qiskit/qiskit-aer -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/home/mtreinish/git/qiskit/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/home/mtreinish/git/qiskit/qiskit-aer/test-dist-err/bin/python -DPYTHON_VERSION_STRING:STRING=3.8.5 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:FILEPATH=/usr/lib/libpython3.8.so -DSKBUILD:BOOL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/mtreinish/git/qiskit/qiskit-aer/test-dist-err/lib/python3.8/site-packages/skbuild/resources/cmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 -DCMAKE_BUILD_TYPE:STRING=Release
  Source directory:
    /home/mtreinish/git/qiskit/qiskit-aer
  Working directory:
    /home/mtreinish/git/qiskit/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build
Please see CMake's output for more information.

Looking at this error my assumption is that scikit build isn't handling the setup_requires arg early enough and numpy isn't getting installed in time for cmake to look for the headers. Normally setuptools.setup() would pull the setup_requires and install the requirements early in the process so that numpy would be present for when extensions are built.

Steps to reproduce the problem

Run python setup.py install in a clean environment without anything installed.

What is the expected behavior?

Aer builds and installs correctly

Suggested solutions

Figure out why numpy isn't getting installed and fix it.

JanLahmann commented 4 years ago

@mtreinish , fyi, when I tried to install on a Raspberry Pi, I needed to install the following packages (plus Rust and libcint) before qiskit to make it work. 

pip install --prefer-binary retworkx pyscf cython six==1.14.*
pip install --prefer-binary 'qiskit[visualization]==0.20.*'

(See https://gist.github.com/JanLahmann/5ef71cfc11e3a08f9edc58ef94c692e0 for an install script that work on a RPi.)

atilag commented 4 years ago

I see that CMake is taking Ninja as the default build tool, we haven't tested builds with Ninja at all (even though it might work). Can you test this?

$ rm -rf _skbuild
$ python setup.py install -- -G"Unix Makefiles"
mtreinish commented 4 years ago

@atilag this was on a clean environment. In a fresh venv I ran:

git clone https://github.com/Qiskit/qiskit-aer
cd qiskit-aer
python setup.py install

If we don't test ninja why does it end up being the default build tool? We should have it default to what we know works.

That being said I tested it using the unix makefiles generator and it still failed the same way:

--------------------------------------------------------------------------------
-- Trying "Unix Makefiles" generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
-- The C compiler identification is GNU 10.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 10.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/qiskit-aer/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying "Unix Makefiles" generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /tmp/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build
  Command:
    /tmp/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake /tmp/qiskit-aer -G 'Unix Makefiles' -DCMAKE_INSTALL_PREFIX:PATH=/tmp/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/tmp/test-aer/bin/python -DPYTHON_VERSION_STRING:STRING=3.8.5 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:FILEPATH=/usr/lib/libpython3.8.so -DSKBUILD:BOOL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/test-aer/lib/python3.8/site-packages/skbuild/resources/cmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 -DCMAKE_BUILD_TYPE:STRING=Release

-- The CXX compiler identification is GNU 10.2.0
-- The C compiler identification is GNU 10.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Performing Test flag_supported
-- Performing Test flag_supported - Success
-- Conan: Automatic detection of conan settings from cmake
-- Conan: Settings= -s;build_type=Release;-s;compiler=gcc;-s;compiler.version=10;-s;compiler.libcxx=libstdc++11
-- Conan: checking conan executable
-- Conan: Found program /tmp/test-aer/bin/conan
-- Conan: Version found Conan version 1.28.2

-- Conan executing: /tmp/test-aer/bin/conan install . -s build_type=Release -s compiler=gcc -s compiler.version=10 -s compiler.libcxx=libstdc++11 -e=CONAN_CMAKE_PROGRAM=/tmp/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake -g=cmake --build=missing
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
CONAN_CMAKE_PROGRAM=/tmp/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake
conanfile.txt: Installing package
Requirements
    fmt/6.2.0 from 'conan-center' - Cache
    muparserx/4.0.8 from 'conan-center' - Cache
    nlohmann_json/3.1.1 from 'conan-center' - Cache
    spdlog/1.5.0 from 'conan-center' - Cache
Packages
    fmt/6.2.0:b173bbda18164d49a449ffadc1c9e817f49e819d - Cache
    muparserx/4.0.8:b173bbda18164d49a449ffadc1c9e817f49e819d - Cache
    nlohmann_json/3.1.1:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    spdlog/1.5.0:5ec35c7aba5410ef8b762c324eb39c457e8e6fe3 - Cache

Installing (downloading, building) binaries...
fmt/6.2.0: Already installed!
muparserx/4.0.8: Already installed!
nlohmann_json/3.1.1: Already installed!
spdlog/1.5.0: Already installed!
conanfile.txt: Generator cmake created conanbuildinfo.cmake
conanfile.txt: Generator txt created conanbuildinfo.txt
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo
-- Conan: Loading conanbuildinfo.cmake
-- Conan: Adjusting output directories
-- Conan: Using cmake targets configuration
-- Library spdlog found /home/mtreinish/.conan/data/spdlog/1.5.0/_/_/package/5ec35c7aba5410ef8b762c324eb39c457e8e6fe3/lib/libspdlog.a
-- Library muparserx found /home/mtreinish/.conan/data/muparserx/4.0.8/_/_/package/b173bbda18164d49a449ffadc1c9e817f49e819d/lib/libmuparserx.a
-- Library fmt found /home/mtreinish/.conan/data/fmt/6.2.0/_/_/package/b173bbda18164d49a449ffadc1c9e817f49e819d/lib/libfmt.a
-- Conan: Adjusting language standard
-- Current conanbuildinfo.cmake directory: /tmp/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build
-- Conan: Compiler GCC>=5, checking major version 10
-- Conan: Checking correct version: 10
-- Looking for OpenMP support...
-- OpenMP found!
-- OpenMP_CXX_FLAGS = -fopenmp
-- OpenMP_EXE_LINKER_FLAGS = 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Looking for OpenBLAS library...
-- BLAS library found: /usr/lib/libopenblas.so
-- No Thrust support enabled
-- Found PythonInterp: /tmp/test-aer/bin/python (found version "3.8.5") 
-- Found PythonLibs: /usr/lib/libpython3.8.so (found version "3.8.5") 
-- Found Cython: /usr/bin/cython  
CMake Error at .eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find NumPy (missing: NumPy_INCLUDE_DIR) (found version "")
Call Stack (most recent call first):
  .eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  /tmp/test-aer/lib/python3.8/site-packages/skbuild/resources/cmake/FindNumPy.cmake:98 (find_package_handle_standard_args)
  cmake/cython_utils.cmake:4 (find_package)
  qiskit/providers/aer/pulse/qutip_extra_lite/cy/CMakeLists.txt:2 (include)

-- Configuring incomplete, errors occurred!
See also "/tmp/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build/CMakeFiles/CMakeOutput.log".
See also "/tmp/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "/tmp/test-aer/lib/python3.8/site-packages/skbuild/setuptools_wrap.py", line 582, in setup
    env = cmkr.configure(cmake_args,
  File "/tmp/test-aer/lib/python3.8/site-packages/skbuild/cmaker.py", line 229, in configure
    raise SKBuildError(

An error occurred while configuring with CMake.
  Command:
    /tmp/qiskit-aer/.eggs/cmake-3.18.2.post1-py3.8-linux-x86_64.egg/cmake/data/bin/cmake /tmp/qiskit-aer -G 'Unix Makefiles' -DCMAKE_INSTALL_PREFIX:PATH=/tmp/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/tmp/test-aer/bin/python -DPYTHON_VERSION_STRING:STRING=3.8.5 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:FILEPATH=/usr/lib/libpython3.8.so -DSKBUILD:BOOL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/test-aer/lib/python3.8/site-packages/skbuild/resources/cmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 -DCMAKE_BUILD_TYPE:STRING=Release
  Source directory:
    /tmp/qiskit-aer
  Working directory:
    /tmp/qiskit-aer/_skbuild/linux-x86_64-3.8/cmake-build
Please see CMake's output for more information.
vvilpas commented 3 years ago

Looking at this error my assumption is that scikit build isn't handling the setup_requires arg early enough and numpy isn't getting installed in time for cmake to look for the headers. Normally setuptools.setup() would pull the setup_requires and install the requirements early in the process so that numpy would be present for when extensions are built.

I think you are right. It looks like requirements are installed after cmake process finish. I think the only thing we can do is check at the beginning of setup.py if numpy is installed and install it if it is not (as we do with scikit-build).