search

trueagi-io / hyperon-experimental

MeTTa programming language implementation
https://metta-lang.dev
MIT License
151 stars 49 forks source link

Build problem (no docker) C extension: conan delegating pybind11 build to cmake which conflicts with pyenv. #749

Closed gl-yziquel closed 3 months ago

gl-yziquel commented 4 months ago

Hi.

I don't quite know how to trace it down much, so I'll say it as it is, and welcome some interaction in order to zoom in better on the problem.

I'm trying to build hyperon metta without docker. (I really do not like docker when used as a build system).

The C extension must be built with the "cmake" followed by "make" commands. This, in turn, seems to delegate to conan. Which tries to build pybind11. For that build of pybind11, conan delegates to cmake. Which, in turn, attempts to detect python.

There is a somewhat well-known cmake / pyenv incompatibility, which I'm told is solved by either passing the -DPYTHON_EXECUTABLE=$(python -c "import sys; print(sys.executable)") setting to the cmake invocation, or altering it directly in the CMakeCache.txt file.

However, as much as I'd like to, I cannot alter that easily (or at least I do not know how) as this cmake build is not the top cmake build but a cmake build of pybind11 delegated along a chain like cmake -> conan -> cmake.

This makes it very hard to build Hyperon MeTTa in some custom environment from source. Which is what I need to do. And a blocker in my adoption of MeTTa (as I keep coming back month over month over such build issues, without finding the time to fix it completely among all the multitasking I'm undertaking).

gl-yziquel commented 4 months ago

To be more specific: I believe the issue occurs since, for other reasons related to other projects, I installed some python3.6 version via pyenv. The cmake at the bottom of the cmake -> conan -> cmake invocation chain now attempts to use 3.6. As python3.6 is an executable but some empty stub that does nothing unless activated, it likely tries to do something like python3.6 --version and fails.

It is a shame that it seems impossible to control the PYTHON_EXECUTABLE or whatever that is at the bottom of the cmake -> conan -> cmake invocation chain to constrain it to use the activated pyenv python executable.

I removed all pyenv 3.6 versions, and it builds fine. But it should be able to build with 3.6 versions hanging around if they are not activated, which is not the case.

vsbogd commented 4 months ago

To me this behavior looks like PyEnv/pybind11 incompatibility https://github.com/pybind/pybind11/issues/2154. Anyway I agree that it is a bug and should be fixed.

@gl-yziquel which CMake version do you use?

gl-yziquel commented 4 months ago

To me this behavior looks like PyEnv/pybind11 incompatibility pybind/pybind11#2154. Anyway I agree that it is a bug and should be fixed.

@gl-yziquel which CMake version do you use?

I tried quite a lot of versions. 3.19.8, 3.26.6 and git HEAD. Don't remember which one, but all failed.

vsbogd commented 4 months ago

Confirm, I can reproduce it on a clean system (CMake version 3.25.1 which is fresh enough). Steps to reproduce:

Build fails with the following log:

pybind11/2.10.1: CMake command: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/home/user/.conan/data/pybind11/2.10.1/_/_/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/build-release/conan/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/home/user/.conan/data/pybind11/2.10.1/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/home/user/.conan/data/pybind11/2.10.1/_/_/source/src"
-- Using Conan toolchain: /home/user/.conan/data/pybind11/2.10.1/_/_/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/build-release/conan/conan_toolchain.cmake
-- The CXX compiler identification is GNU 12.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- pybind11 v2.10.1 
-- CMake 3.25.1
-- Found PythonInterp: /home/user/.pyenv/shims/python3.6 (Required is at least version "3.6") 
CMake Error at tools/FindPythonLibsNew.cmake:147 (message):
  Python config failure:

  pyenv: python3.6: command not found

  The `python3.6' command exists in these Python versions:

    3.6.15

  Note: See 'pyenv help global' for tips on allowing both

        python2 and python3 to be found.

Call Stack (most recent call first):
  tools/pybind11Tools.cmake:50 (find_package)
  tools/pybind11Common.cmake:180 (include)
  CMakeLists.txt:206 (include)

-- Configuring incomplete, errors occurred!
See also "/home/user/.conan/data/pybind11/2.10.1/_/_/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/build-release/CMakeFiles/CMakeOutput.log".
pybind11/2.10.1: 
pybind11/2.10.1: ERROR: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' build failed
pybind11/2.10.1: WARN: Build folder /home/user/.conan/data/pybind11/2.10.1/_/_/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/build-release
ERROR: pybind11/2.10.1: Error in build() method, line 41
    cmake.configure()
    ConanException: Error 1 while executing cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/home/user/.conan/data/pybind11/2.10.1/_/_/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/build-release/conan/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/home/user/.conan/data/pybind11/2.10.1/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/home/user/.conan/data/pybind11/2.10.1/_/_/source/src"
CMake Error at CMakeLists.txt:31 (include):
  include could not find requested file:

    /home/user/hyperon-experimental/build/hyperonpy-prefix/src/hyperonpy-build/conan_paths.cmake

-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "3.0.13")  
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.13") 
CMake Error at CMakeLists.txt:38 (find_package):
  By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "pybind11",
  but CMake did not find one.

  Could not find a package configuration file provided by "pybind11" with any
  of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

  Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
  "pybind11_DIR" to a directory containing one of the above files.  If
  "pybind11" provides a separate development package or SDK, be sure it has
  been installed.

-- Configuring incomplete, errors occurred!
See also "/home/user/hyperon-experimental/build/hyperonpy-prefix/src/hyperonpy-build/CMakeFiles/CMakeOutput.log".
make[2]: *** [CMakeFiles/hyperonpy.dir/build.make:101: hyperonpy-prefix/src/hyperonpy-stamp/hyperonpy-configure] Error 1
make[1]: *** [CMakeFiles/Makefile2:112: CMakeFiles/hyperonpy.dir/all] Error 2
make: *** [Makefile:101: all] Error 2
gl-yziquel commented 4 months ago

Confirm, I can reproduce it on a clean system (CMake version 3.25.1 which is fresh enough). Steps to reproduce:

This is indeed the precise error messge that occurs when python 3.6 lies around.

vsbogd commented 3 months ago

@gl-yziquel I migrated build to the Conan 2 (see #756) because I thought it will help making build more transparent. But it seems it is still not clear how to pass PYTHON_EXECUTABLE to pybind11 build script. I will try to fix it after migration PR is merged.

vsbogd commented 3 months ago

Looks like it is fixed after #756. Cannot reproduce it anymore. @gl-yziquel I would propose closing it and re-open if problem raises again.