Problem with building with cmake - does not find correct python3 executable

[stefanarridge]

When trying to build on the cmake_new-output branch, cmake exits because it is not able to import yt for a particular python3 executable. This is in spite of the fact that I run cmake while having a conda environment activated (in which yt is installed). This is because it only finds /usr/bin/python3.8 rather than the appropriate python executable for the environment (which is found at /home/stefan/yt-conda/envs/enzo-e-analysis/bin/python for my set-up, and is python version 3.10.1). The version of cmake I am using is 3.16.3.

Here is the full output from my terminal:

$ cmake -DCHARM_ROOT=~/codes/charm_v_7_0_0/build-mpi -DEnzo-E_CONFIG=linux_gcc -DUSE_GRACKLE=ON -DGrackle_ROOT=~/local/grackle ..
-- Using machine configuration file from /home/stefan/codes/enzo-e/config/linux_gcc.cmake
-- Loading machine configuration for generic Linux machine with GCC stack.

-- Charm++ built in non-SMP mode
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
CMake Error at cmake/PythonModuleCheck.cmake:32 (message):
  Required python module(s) yt not found.
Call Stack (most recent call first):
  CMakeLists.txt:296 (required_python_modules_found)

-- Configuring incomplete, errors occurred!
See also "/home/stefan/codes/enzo-e/build_mpi/CMakeFiles/CMakeOutput.log".
See also "/home/stefan/codes/enzo-e/build_mpi/CMakeFiles/CMakeError.log".

Here are the full log files: CMakeError.log CMakeOutput.log

I have managed to get around this by adding the argument -DPython3_EXECUTABLE=/home/stefan/yt-conda/envs/enzo-e-analysis/bin/python to the cmake command, although not sure if this is the ideal solution. In any case, this information should be useful in case anyone else encounters the same problem.

[pgrete]

It looks like you tried to call cmake again in a directory with an existing CMakeCache.txt.

Could you please try a fresh configure (i.e., for a new build directory) and using the -DPython3_FIND_VIRTUALENV=ONLY option? For example (from the main enzo-e directory): cmake -DCHARM_ROOT=~/codes/charm_v_7_0_0/build-mpi -DEnzo-E_CONFIG=linux_gcc -DUSE_GRACKLE=ON -DGrackle_ROOT=~/local/grackle -DPython3_FIND_VIRTUALENV=ONLY -Bbuild-test

[stefanarridge]

I just did what you suggested, here is the terminal output:

$ cmake -DCHARM_ROOT=~/codes/charm_v_7_0_0/build-mpi -DEnzo-E_CONFIG=linux_gcc -DUSE_GRACKLE=ON -DGrackle_ROOT=~/local/grackle -DPython3_FIND_VIRTUALENV=ONLY -Bbuild-test
-- Using machine configuration file from /home/stefan/codes/enzo-e/config/linux_gcc.cmake
-- Loading machine configuration for generic Linux machine with GCC stack.

-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- The Fortran compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/g++
-- Check for working CXX compiler: /usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working Fortran compiler: /usr/bin/gfortran
-- Check for working Fortran compiler: /usr/bin/gfortran  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/bin/gfortran supports Fortran 90
-- Checking whether /usr/bin/gfortran supports Fortran 90 -- yes
-- Setting build type to 'Release' as none was specified.
-- Found Charm: /home/stefan/codes/charm_v_7_0_0/build-mpi/bin/charmc  
-- Looking for include file /home/stefan/codes/charm_v_7_0_0/build-mpi/include/conv-mach-opt.h
-- Looking for include file /home/stefan/codes/charm_v_7_0_0/build-mpi/include/conv-mach-opt.h - found
-- Looking for CMK_SMP
-- Looking for CMK_SMP - not found
-- Charm++ built in non-SMP mode
-- Found Grackle: /home/stefan/local/grackle/lib/libgrackle.so  
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found version "1.71.0") found components: filesystem 
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
-- Found PNG: /usr/lib/x86_64-linux-gnu/libpng.so (found version "1.6.37") 
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.4") found components: C 
-- Found Python3: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter 
CMake Error at cmake/PythonModuleCheck.cmake:32 (message):
  Required python module(s) yt not found.
Call Stack (most recent call first):
  CMakeLists.txt:296 (required_python_modules_found)

-- Configuring incomplete, errors occurred!
See also "/home/stefan/codes/enzo-e/build-test/CMakeFiles/CMakeOutput.log".
See also "/home/stefan/codes/enzo-e/build-test/CMakeFiles/CMakeError.log".

Here are the log files:

CMakeError.log CMakeOutput.log

[mabruzzo]

It looks like you're using CMake version 3.16.3.

This documentation suggests that the -DPython3_FIND_VIRTUALENV=ONLY hint only supports conda environments starting in CMake version >= 3.17 (in previous versions, it only supported environments managed by virtualenv).

As an aside, apt only natively installs CMake 3.16 on Ubuntu 20.04. So I think this could be a common problem. With that in mind, I think we should probably consider customizing how we have CMake search for the python interpreter...

[pgrete]

I agree that we should cover that case as Ubuntu is common setup.

How about setting 3.16 as minimum version (so that we get support for setting the Python_EXECUTABLE directly and would not need any additional logic on our part)?

[stefanarridge]

How about setting 3.16 as minimum version

Did you mean "setting 3.17".

[pgrete]

How about setting 3.16 as minimum version

Did you mean "setting 3.17".

No, I meant 3.16 as this already allow us to use Python_EXECUTABLE. 3.17 would be preferable so that, as @mabruzzo pointed out, Python3_FIND_VIRTUALENV=ONLY would also work with conda, but given that 3.16 is the default for Ubuntu 20.04 (and thus may be widely used), 3.16 seems to be a "not too old but still providing a workable solution" option.

[stefanarridge]

Ok, I see what you mean.

[mabruzzo]

One thing that might be worth considering is doing something like:

if (USE_PYTHON_BINARY_IN_PATH)
  execute_process(COMMAND python -c "import sys; print(sys.executable)"
                  OUTPUT_VARIABLE Python_EXECUTABLE ERROR_QUIET)
endif()

Alternatively, it might be just as useful to add a note to the documentation encouraging people to execute python -c "import sys; print(sys.executable)" with their preferred python installation, to figure out what to assign to the Python_EXECUTABLE variable.

[stefanarridge]

@mabruzzo: what's the difference between python -c "import sys; print(sys.executable)" and which python?

[mabruzzo]

In the majority of cases, they're equivalent. However, python -c "import sys; print(sys.executable)" is somewhat more portable.

A concrete example is when you use pyenv to manage multiple python installations (like I do). pyenv works by providing shims for various python commands. These shims are executable shell scripts that simply forward arguments to the correct executable.

• Calling which python or command -v python will just return the path to the shim for the python executable: $(pyenv_root)/shims/python
• Calling python -c "import sys; print(sys.executable)" actually returns the correct path

[pgrete]

I just updated the logic and added a more verbose error message. The output may now look like

-- Found HDF5: /usr/lib/libhdf5.so;/usr/lib/libsz.so;/usr/lib/libz.so;/usr/lib/libdl.a;/usr/lib/libm.so (found version "1.12.1") found components: C 
-- Found Python3: /home/pgrete/venv/bin/python3.10 (found version "3.10.4") found components: Interpreter 
--    IMPORTANT: Make sure this is the Python version you want to use. If not, e.g., because you want to use a conda or virtualenv based version, try setting `-DPython3_FIND_VIRTUALENV=ONLY` or `-DPython3_EXECUTABLE=/path/to/desired/python`.
CMake Error at cmake/PythonModuleCheck.cmake:32 (message):
  Required python module(s) yt not found.
Call Stack (most recent call first):
  CMakeLists.txt:316 (required_python_modules_found)

Note that the "IMPORTANT" line will always be shown as I expect there will be fewer (unintended) errors here if we are more verbose about it.