search

DeepLearnPhysics / larcv3

Third version of larcv. This is a complete replacement for larcv2.
MIT License
11 stars 6 forks source link

Pip install not working #52

Closed bear-is-asleep closed 2 years ago

bear-is-asleep commented 2 years ago

I'm trying to run pip install larcv and I am running to an error in the build stage. I get the same error message when I try building from the source.

pip3.9 install larcv

Defaulting to user installation because normal site-packages is not writeable
Collecting larcv
  Using cached larcv-3.4.1.tar.gz (7.9 MB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /home/bc1/.local/lib/python3.9/site-packages (from larcv) (1.21.6)
Requirement already satisfied: scikit-build in /home/bc1/.local/lib/python3.9/site-packages (from larcv) (0.14.1)
Requirement already satisfied: wheel>=0.29.0 in /home/bc1/.local/lib/python3.9/site-packages (from scikit-build->larcv) (0.37.1)
Requirement already satisfied: setuptools>=28.0.0 in /usr/local/lib/python3.9/site-packages (from scikit-build->larcv) (58.1.0)
Requirement already satisfied: packaging in /home/bc1/.local/lib/python3.9/site-packages (from scikit-build->larcv) (21.3)
Requirement already satisfied: distro in /home/bc1/.local/lib/python3.9/site-packages (from scikit-build->larcv) (1.7.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/bc1/.local/lib/python3.9/site-packages (from packaging->scikit-build->larcv) (3.0.8)
Building wheels for collected packages: larcv
error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [196 lines of output]
      /usr/local/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'use_scm_version'
        warnings.warn(msg)

      --------------------------------------------------------------------------------
      -- Trying "Ninja" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
      -- Configuring incomplete, errors occurred!
      See also "/tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "Ninja" generator - failure
      --------------------------------------------------------------------------------

      --------------------------------------------------------------------------------
      -- Trying "Unix Makefiles" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      -- The C compiler identification is GNU 7.5.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is GNU 7.5.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done
      -- Generating done
      -- Build files have been written to: /tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/_cmake_test_compile/build
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "Unix Makefiles" generator - success
      --------------------------------------------------------------------------------

      Configuring Project
        Working directory:
          /tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/_skbuild/linux-x86_64-3.9/cmake-build
        Command:
          cmake /tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/src -G 'Unix Makefiles' -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/_skbuild/linux-x86_64-3.9/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/python3.9 -DPYTHON_VERSION_STRING:STRING=3.9.10 -DPYTHON_INCLUDE_DIR:PATH=/usr/local/include/python3.9 -DPYTHON_LIBRARY:FILEPATH=/usr/local/lib/libpython3.9.a -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/bc1/.local/lib/python3.9/site-packages/skbuild/resources/cmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 -DCMAKE_PYTHON_BINDINGS=ON -DMPI:BOOL=OFF -DOPENMP:BOOL=OFF -DDOCS:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release

      -- The C compiler identification is GNU 7.5.0
      -- The CXX compiler identification is GNU 7.5.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.0.1")
      -- Found PythonLibs: /usr/local/lib/libpython3.9.a (found version "3.9.10")
      1
      /home/bc1/.local/lib/python3.9/site-packages/numpy/core/include
      -- Using the single-header code from /tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/src/json/single_include/
      NLOHMANN_JSON_INCLUDE_INSTALL_DIR:
      Adding pybind11
      -- pybind11 v2.9.0
      -- Found PythonInterp: /usr/local/bin/python3.9 (found version "3.9.10")
      -- Found PythonLibs: /usr/local/lib/libpython3.9.a
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      Using hdf5 parallel: FALSE
      -- Configuring done
      -- Generating done
      CMake Warning:
        Manually-specified variables were not used by the project:

          SKBUILD

      -- Build files have been written to: /tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/_skbuild/linux-x86_64-3.9/cmake-build
      [  1%] Building CXX object larcv3/app/sbnd_imagemod/CMakeFiles/sbnd_imagemod.dir/CompressAndEmbed.cxx.o

...........................

      [ 96%] Linking CXX shared library liblarcv3.so
      /usr/bin/ld: /usr/local/lib/libpython3.9.a(abstract.o): relocation R_X86_64_PC32 against symbol `PyObject_SetItem' can not be used when making a shared object; recompile with -fPIC
      /usr/bin/ld: final link failed: Bad value
      collect2: error: ld returned 1 exit status
      larcv3/CMakeFiles/larcv3.dir/build.make:201: recipe for target 'larcv3/liblarcv3.so' failed
      make[2]: *** [larcv3/liblarcv3.so] Error 1
      CMakeFiles/Makefile2:314: recipe for target 'larcv3/CMakeFiles/larcv3.dir/all' failed
      make[1]: *** [larcv3/CMakeFiles/larcv3.dir/all] Error 2
      Makefile:135: recipe for target 'all' failed
      make: *** [all] Error 2
      Traceback (most recent call last):
        File "/home/bc1/.local/lib/python3.9/site-packages/skbuild/setuptools_wrap.py", line 645, in setup
          cmkr.make(make_args, install_target=cmake_install_target, env=env)
        File "/home/bc1/.local/lib/python3.9/site-packages/skbuild/cmaker.py", line 660, in make
          self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
        File "/home/bc1/.local/lib/python3.9/site-packages/skbuild/cmaker.py", line 684, in make_impl
          raise SKBuildError(

      An error occurred while building with CMake.
        Command:
          cmake --build . --target install --config Release --
        Install target:
          install
        Source directory:
          /tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de
        Working directory:
          /tmp/pip-install-nivbg9d4/larcv_ab63dc0365694d808ad63c88c984d2de/_skbuild/linux-x86_64-3.9/cmake-build
      Please check the install target is valid and see CMake's output for more information.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for larcv

I'm running on Linux Mint 19.1 Cinnamon 4.0.10. Thanks in advance!

coreyjadams commented 2 years ago

Hi,

You've got a new error I haven't seen in larcv before! -fPIC means Position Independent Code which usually is included by default.

Here's the heart of the problem:

      /usr/bin/ld: /usr/local/lib/libpython3.9.a(abstract.o): relocation R_X86_64_PC32 against symbol `PyObject_SetItem' can not be used when making a shared object; recompile with -fPIC

It looks like there is an error linking against a python call. What python are you using? You've got python installed as a static library instead of dynamic library. The difference is summarized here by the way.

Here's a plausible fix: If you don't mind, can you clone the repo and change this line locally?

https://github.com/DeepLearnPhysics/larcv3/blob/develop/src/CMakeLists.txt#L7

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -O3 -g")

to

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -O3 -g -fPIC")

Then install with:

python setup.py build -j 32 # The error will happen here, if it happens.
python setup.py install

I've verified this doesn't break things on my laptop, hopefully it will fix things on yours. Please let me know how it goes.

bear-is-asleep commented 2 years ago

Hi Corey,

Thanks very much for the quick response! I just tried this, and I'm still getting the same error: python setup.py build -j 32

/usr/local/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'use_scm_version'
  warnings.warn(msg)
[  6%] Built target sbnd_imagemod
[ 16%] Built target base
[ 20%] Built target filter
[ 26%] Built target processor
[ 43%] Built target imagemod
[ 68%] Built target queueio
[ 95%] Built target dataformat
[ 96%] Linking CXX shared library liblarcv3.so
/usr/bin/ld: /usr/local/lib/libpython3.9.a(abstract.o): relocation R_X86_64_PC32 against symbol `PyObject_SetItem' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
larcv3/CMakeFiles/larcv3.dir/build.make:201: recipe for target 'larcv3/liblarcv3.so' failed
make[2]: *** [larcv3/liblarcv3.so] Error 1
CMakeFiles/Makefile2:314: recipe for target 'larcv3/CMakeFiles/larcv3.dir/all' failed
make[1]: *** [larcv3/CMakeFiles/larcv3.dir/all] Error 2
Makefile:135: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "/home/bc1/.local/lib/python3.9/site-packages/skbuild/setuptools_wrap.py", line 645, in setup
    cmkr.make(make_args, install_target=cmake_install_target, env=env)
  File "/home/bc1/.local/lib/python3.9/site-packages/skbuild/cmaker.py", line 660, in make
    self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
  File "/home/bc1/.local/lib/python3.9/site-packages/skbuild/cmaker.py", line 684, in make_impl
    raise SKBuildError(

An error occurred while building with CMake.
  Command:
    cmake --build . --target install --config Release -- -j 32
  Install target:
    install
  Source directory:
    /home/bc1/python_utils/larcv3
  Working directory:
    /home/bc1/python_utils/larcv3/_skbuild/linux-x86_64-3.9/cmake-build
Please check the install target is valid and see CMake's output for more information.

Do I need to clone into a specific location? I've just been doing it in my home directory. Also, I did not know python could be installed statically, nor do I know how to fix it (google didn't turn anything up).

Here's some other useful commands that may help:

python -V
Python 3.9.10
which python
/usr/bin/python
bear-is-asleep commented 2 years ago

Here's the first few lines of my cmake file:

cmake_minimum_required(VERSION 3.1 FATAL_ERROR)

# general configuration for compilation:
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -O3 -g")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -O3 -g -fPIC")
coreyjadams commented 2 years ago

So, that's a bummer, though of course it wasn't that easy.

How did you install python? I've spun up a Mint 19.1 image and the default python is 2.7, python3 is 3.6.7.

When you build from source, it prints a bunch of cmake info before compiling. Could you share that here, please?

bear-is-asleep commented 2 years ago

Hi Corey,

To be honest, I'm not quite sure about the method I installed python3, I could reinstall it if you recommend a better method. If I were to hazard a guess, I was testing installing python locally on my machine, so I probably used the following method: https://notes.webutvikling.org/get-python-virtualenv-pip-without-sudo/

Here's the cmake info:

python3.9 setup.py build -j64
/usr/local/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'use_scm_version'
  warnings.warn(msg)

--------------------------------------------------------------------------------
-- Trying "Ninja" generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
-- Configuring incomplete, errors occurred!
See also "/home/bc1/larcv3/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying "Ninja" generator - failure
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
-- Trying "Unix Makefiles" generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
-- The C compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 7.5.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bc1/larcv3/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying "Unix Makefiles" generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /home/bc1/larcv3/_skbuild/linux-x86_64-3.9/cmake-build
  Command:
    cmake /home/bc1/larcv3/src -G 'Unix Makefiles' -DCMAKE_INSTALL_PREFIX:PATH=/home/bc1/larcv3/_skbuild/linux-x86_64-3.9/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/python3.9 -DPYTHON_VERSION_STRING:STRING=3.9.10 -DPYTHON_INCLUDE_DIR:PATH=/usr/local/include/python3.9 -DPYTHON_LIBRARY:FILEPATH=/usr/local/lib/libpython3.9.a -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/bc1/.local/lib/python3.9/site-packages/skbuild/resources/cmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 -DCMAKE_PYTHON_BINDINGS=ON -DMPI:BOOL=OFF -DOPENMP:BOOL=OFF -DDOCS:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release

-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.0.1")  
-- Found PythonLibs: /usr/local/lib/libpython3.9.a (found version "3.9.10") 
1
/home/bc1/.local/lib/python3.9/site-packages/numpy/core/include
-- Using the single-header code from /home/bc1/larcv3/src/json/single_include/
NLOHMANN_JSON_INCLUDE_INSTALL_DIR: 
Adding pybind11
-- pybind11 v2.9.0 
-- Found PythonInterp: /usr/local/bin/python3.9 (found version "3.9.10") 
-- Found PythonLibs: /usr/local/lib/libpython3.9.a
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
Using hdf5 parallel: FALSE
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    SKBUILD
    -- Build files have been written to: /home/bc1/larcv3/_skbuild/linux-x86_64-3.9/cmake-build
[  3%] Building CXX object larcv3/core/processor/CMakeFiles/processor.dir/ProcessBase.cxx.o

. .

Thanks again for the help! Bear

coreyjadams commented 2 years ago

Ah, so, you built it yourself! Great I am feeling more hopeful. I've just pushed some changes to larcv, in particular it ought to let you build against a different python version as a test. We can try two paths: build against the system python3, which ought to be OK. Or, you can recompile python as a shared library.

Option A: rebuild python. Follow the steps you posted, since they mostly seemed to work, but try this command:

./configure --enable-shared -prefix=/home/username/python/python27
# Instead of this:
# ./configure -prefix=/home/username/python/python27

Option B: Build against system python:

Can you try it with the system python install at /usr/bin/python3/? On mint 19.1 I bet it's around python 3.7 or so. I just made a push to larcv to build against exactly the python you call setup.py with - it's not on pypi yet so build from source.

You can set up the deps in the local area (pip3 install --user numpy etc).

Make sure that the build output doesn't have this type of stuff:

-- Found PythonInterp: /usr/local/bin/python3.9 (found version "3.9.10") 
-- Found PythonLibs: /usr/local/lib/libpython3.9.a # This is a static library!!

Consider that you built your own python, you may want to recompile Python anyways. Or, you could try the one from apt? Mint has a pretty good package manager and I've never had any issues with it.

bear-is-asleep commented 2 years ago

Option A worked! I should add that I had to do a few other things since I installed python3.10.4 locally:

-pip3.10 install cython -snap install cmake (requires cmake > 3.12)

Otherwise, it's installed. Thanks for your help!

coreyjadams commented 2 years ago

Awesome! cython as a dependency surprises me, it's not used in larcv3, but sounds good.

Yes to cmake - it's a build time dependency. Glad it's working. I'll close the issue. For anyone in the future, the main issue seemed to be:

Solution was: rebuilding python with --enable-shared.