Closed stefanarridge closed 2 years ago
It looks like you tried to call cmake
again in a directory with an existing CMakeCache.txt
.
Could you please try a fresh configure (i.e., for a new build directory) and using the -DPython3_FIND_VIRTUALENV=ONLY
option?
For example (from the main enzo-e directory):
cmake -DCHARM_ROOT=~/codes/charm_v_7_0_0/build-mpi -DEnzo-E_CONFIG=linux_gcc -DUSE_GRACKLE=ON -DGrackle_ROOT=~/local/grackle -DPython3_FIND_VIRTUALENV=ONLY -Bbuild-test
I just did what you suggested, here is the terminal output:
$ cmake -DCHARM_ROOT=~/codes/charm_v_7_0_0/build-mpi -DEnzo-E_CONFIG=linux_gcc -DUSE_GRACKLE=ON -DGrackle_ROOT=~/local/grackle -DPython3_FIND_VIRTUALENV=ONLY -Bbuild-test
-- Using machine configuration file from /home/stefan/codes/enzo-e/config/linux_gcc.cmake
-- Loading machine configuration for generic Linux machine with GCC stack.
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- The Fortran compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/g++
-- Check for working CXX compiler: /usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working Fortran compiler: /usr/bin/gfortran
-- Check for working Fortran compiler: /usr/bin/gfortran -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/bin/gfortran supports Fortran 90
-- Checking whether /usr/bin/gfortran supports Fortran 90 -- yes
-- Setting build type to 'Release' as none was specified.
-- Found Charm: /home/stefan/codes/charm_v_7_0_0/build-mpi/bin/charmc
-- Looking for include file /home/stefan/codes/charm_v_7_0_0/build-mpi/include/conv-mach-opt.h
-- Looking for include file /home/stefan/codes/charm_v_7_0_0/build-mpi/include/conv-mach-opt.h - found
-- Looking for CMK_SMP
-- Looking for CMK_SMP - not found
-- Charm++ built in non-SMP mode
-- Found Grackle: /home/stefan/local/grackle/lib/libgrackle.so
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found version "1.71.0") found components: filesystem
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Found PNG: /usr/lib/x86_64-linux-gnu/libpng.so (found version "1.6.37")
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.4") found components: C
-- Found Python3: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter
CMake Error at cmake/PythonModuleCheck.cmake:32 (message):
Required python module(s) yt not found.
Call Stack (most recent call first):
CMakeLists.txt:296 (required_python_modules_found)
-- Configuring incomplete, errors occurred!
See also "/home/stefan/codes/enzo-e/build-test/CMakeFiles/CMakeOutput.log".
See also "/home/stefan/codes/enzo-e/build-test/CMakeFiles/CMakeError.log".
Here are the log files:
It looks like you're using CMake version 3.16.3.
This documentation suggests that the -DPython3_FIND_VIRTUALENV=ONLY
hint only supports conda
environments starting in CMake version >= 3.17 (in previous versions, it only supported environments managed by virtualenv
).
As an aside, apt only natively installs CMake 3.16 on Ubuntu 20.04. So I think this could be a common problem. With that in mind, I think we should probably consider customizing how we have CMake search for the python interpreter...
I agree that we should cover that case as Ubuntu is common setup.
How about setting 3.16 as minimum version (so that we get support for setting the Python_EXECUTABLE
directly and would not need any additional logic on our part)?
How about setting 3.16 as minimum version
Did you mean "setting 3.17".
How about setting 3.16 as minimum version
Did you mean "setting 3.17".
No, I meant 3.16 as this already allow us to use Python_EXECUTABLE
.
3.17 would be preferable so that, as @mabruzzo pointed out, Python3_FIND_VIRTUALENV=ONLY
would also work with conda, but given that 3.16 is the default for Ubuntu 20.04 (and thus may be widely used), 3.16 seems to be a "not too old but still providing a workable solution" option.
Ok, I see what you mean.
One thing that might be worth considering is doing something like:
if (USE_PYTHON_BINARY_IN_PATH)
execute_process(COMMAND python -c "import sys; print(sys.executable)"
OUTPUT_VARIABLE Python_EXECUTABLE ERROR_QUIET)
endif()
Alternatively, it might be just as useful to add a note to the documentation encouraging people to execute python -c "import sys; print(sys.executable)"
with their preferred python installation, to figure out what to assign to the Python_EXECUTABLE
variable.
@mabruzzo: what's the difference between python -c "import sys; print(sys.executable)"
and which python
?
In the majority of cases, they're equivalent. However, python -c "import sys; print(sys.executable)"
is somewhat more portable.
A concrete example is when you use pyenv
to manage multiple python installations (like I do). pyenv
works by providing shims for various python commands. These shims are executable shell scripts that simply forward arguments to the correct executable.
which python
or command -v python
will just return the path to the shim for the python executable: $(pyenv_root)/shims/python
python -c "import sys; print(sys.executable)"
actually returns the correct pathI just updated the logic and added a more verbose error message. The output may now look like
-- Found HDF5: /usr/lib/libhdf5.so;/usr/lib/libsz.so;/usr/lib/libz.so;/usr/lib/libdl.a;/usr/lib/libm.so (found version "1.12.1") found components: C
-- Found Python3: /home/pgrete/venv/bin/python3.10 (found version "3.10.4") found components: Interpreter
-- IMPORTANT: Make sure this is the Python version you want to use. If not, e.g., because you want to use a conda or virtualenv based version, try setting `-DPython3_FIND_VIRTUALENV=ONLY` or `-DPython3_EXECUTABLE=/path/to/desired/python`.
CMake Error at cmake/PythonModuleCheck.cmake:32 (message):
Required python module(s) yt not found.
Call Stack (most recent call first):
CMakeLists.txt:316 (required_python_modules_found)
Note that the "IMPORTANT" line will always be shown as I expect there will be fewer (unintended) errors here if we are more verbose about it.
When trying to build on the
cmake_new-output
branch, cmake exits because it is not able to import yt for a particular python3 executable. This is in spite of the fact that I run cmake while having a conda environment activated (in which yt is installed). This is because it only finds/usr/bin/python3.8
rather than the appropriate python executable for the environment (which is found at/home/stefan/yt-conda/envs/enzo-e-analysis/bin/python
for my set-up, and is python version 3.10.1). The version of cmake I am using is 3.16.3.Here is the full output from my terminal:
Here are the full log files: CMakeError.log CMakeOutput.log
I have managed to get around this by adding the argument
-DPython3_EXECUTABLE=/home/stefan/yt-conda/envs/enzo-e-analysis/bin/python
to the cmake command, although not sure if this is the ideal solution. In any case, this information should be useful in case anyone else encounters the same problem.