Open xEnVrE opened 3 years ago
traversaro
commented
3 years ago Just to understand a bit more your request, can you explain a concrete scenario in which you would use this feature. In particular how do you plan to install multiple version of Python on the same system? Is that something supported in the system package manager that you are using, or are you installing one Python with your system package manager, and another version of Python in some other way?
traversaro
commented
3 years ago In any case, have you tried to compile YARP by passing the CMake option -DPython3_FIND_STRATEGY:STRING=LOCATION? As documented in https://cmake.org/cmake/help/git-stage/module/FindPython3.html#hints, this should give you the behavior that you are asking for. Note that the default value for this variable is VERSION, i.e. look for the newest version in any case as you described, but this is regulated by Policy CMP0094 .
xEnVrE
commented
3 years ago Just to understand a bit more your request, can you explain a concrete scenario in which you would use this feature.
When trying/using code associated to papers from the Machine Learning / Computer Vision communities, it happens that the code is compatible with older versions of ML frameworks, e.g. tensorflow and pytorch, only. These versions, in turn, require specific version of Python. Then, if we want to integrate this code in a YARP module the problem arises.
In particular how do you plan to install multiple version of Python on the same system?
E.g., in Ubuntu there are specific, unofficial, ppas from which one can install older versions of Python via apt.
have you tried to compile YARP by passing the CMake option -DPython3_FIND_STRATEGY:STRING=LOCATION
I tried modifying the file https://github.com/robotology/yarp/blob/master/bindings/python/CMakeLists.txt in
find_package(Python3 <version> COMPONENTS Interpreter Development REQUIRED) while passing DPython3_FIND_STRATEGY:STRING=LOCATION but it was still getting the latest version.
Also, I tried to read the description associated to LOCATION and it says "Stops lookup as soon as a version satisfying version constraints is founded. ". However, I am not sure how these "version constraints" can be explicited. If the only way to is to pass <version> in find_package(Python3 <version> ...), then the actual CMakeLists.txt would not allow the user to specify the version as the current call is
Or maybe I misunderstood and there is an alternative way to communicate the version to FindPython.
traversaro
commented
3 years ago Just to understand a bit more your request, can you explain a concrete scenario in which you would use this feature.
When trying/using code associated to papers from the Machine Learning / Computer Vision communities, it happens that the code is compatible with older versions of ML frameworks, e.g. tensorflow and pytorch, only. These versions, in turn, require specific version of Python. Then, if we want to integrate this code in a YARP module the problem arises.
FYI for those specific scenarios you could be interesting in conda, recently we documented how to install robotology-superbuild using conda-forge dependencies (where you can specify the Python version that you use) in https://github.com/robotology/robotology-superbuild/blob/master/doc/conda-forge.md . We are still not testing ROBOTOLOGY_USES_PYTHON option in CI, however. We also plan to eventually provide YARP binaries in a dedicated conda channel, see https://github.com/robotology/robotology-superbuild/issues/620 . On that, it would be interesting on which version of Python users like you could be interesting.
In particular how do you plan to install multiple version of Python on the same system?
E.g., in Ubuntu there are specific, unofficial, ppas from which one can install older versions of Python via
apt.
Can you link one of this PPAs and specify with which Ubuntu version you used it? For example the one that you are using for your tests? To understand how they install Python, which Python is used by default in that case (i.e. python3 in the PATH which version launches?).
However, I am not sure how these "version constraints" can be explicited. If the only way to is to pass
in find_package(Python3 ...), then the actual CMakeLists.txt would not allow the user to specify the version as the current call is
The <version> passed to find_package is indeed the only way to specify the minimum version, but they are not intended to be set by users configuring the project, but just by the project maintainer if indeed the project only works from a given version of Python onward. However, I suggested to use -DPython3_FIND_STRATEGY:STRING=LOCATION only because I tough the Python version that you want to use was the first one that would be found by the LOCATION strategy, but probably I was wrong (and hence the reason why I asked for details on how specifically you are installing Python).
However, if the Python that you want to use is not the "default system one" (whatever it means) I think the only remaining option without modifying YARP CMake's script is to pass explicitly the Python3_EXECUTABLE, Python3_LIBRARY, Python3_INCLUDE_DIR and (if necessary) Python3_NumPy_INCLUDE_DIR CMake variables as described in https://cmake.org/cmake/help/latest/module/FindPython3.html#artifacts-specification . The exact values of this variables depend on how do you installed Python, but I guess that in the case you described Python3_EXECUTABLE should point to /usr/bin/python3.7. I am not sure about the rest of the variables, but it should be easy to understand once we now how you did installed the different Python versions.
xEnVrE
commented
3 years ago We are still not testing ROBOTOLOGY_USES_PYTHON option in CI, however.
If, by following the instructions that you provided, it can be tested safely I can give it a try just to see if the bindings are then provided for the version of python specified in conda
Can you link one of this PPAs and specify with which Ubuntu version you used it?
Actually I have several version of Python but in Arch Linux (using AUR). However, I know that @Arya07 has a similar scenario in Ubuntu. @Arya07 can you maybe specify which ppa did you add on Ubuntu in order to install them?
only because I tough the Python version that you want to use was the first one that would be found by the LOCATION strategy
Indeed this is something that is not easy to control. E.g., in Arch I have all the executables of python, for several versions, in the same place /usr/bin/.
I think the only remaining option without modifying YARP CMake's script is to pass explicitly the Python3_EXECUTABLE, Python3_LIBRARY, Python3_INCLUDE_DIR and (if necessary) Python3_NumPy_INCLUDE_DIR CMake variables as described in
I will try this suggestion since I know how to force the version and I can print the variables afterwards (I hope). Then I can try by setting these variables as you suggested without modifying the YARP CMake script.
xEnVrE
commented
3 years ago I wanted to add that, in the past, this user case was actually supported (or at least it seems to me) inside YARP via the variable YARP_USE_PYTHON_VERSION (now deprecated and not used anymore).
E.g., If I switch back to https://github.com/robotology/yarp/commit/153b045b42372f68a2d8049a4bfd780bfc75508c I can obtain the desired behavior.
cmake <YARP_SRC>/bindings/ -DCREATE_PYTHON=true -DYARP_USE_PYTHON_VERSION=3.7.9 .
(...)
-- [x] Bindings (YARP_COMPILE_BINDINGS)
-- [ ] Java bindings (CREATE_JAVA)
-- [x] Python bindings (CREATE_PYTHON)
-- [ ] Perl bindings (CREATE_PERL)
-- [ ] Chicken Scheme bindings (CREATE_CHICKEN)
-- [ ] CSharp bindings (CREATE_CSHARP)
-- [ ] Allegro Common Lisp bindings (CREATE_ALLEGRO)
-- [ ] TCL bindings (CREATE_TCL)
-- [ ] Ruby bindings (CREATE_RUBY)
-- [ ] Lua bindings (CREATE_LUA)
-- [ ] Octave bindings (CREATE_OCTAVE)
-- Found PythonInterp: /usr/bin/python3.7 (found suitable version "3.7.9", minimum required is "3.7.9")
-- Found PythonLibs: /usr/lib/libpython3.7m.so (found suitable exact version "3.7.9")
(...)However, this old CMakeLists.txt was not using FindPython.
xEnVrE
commented
3 years ago After forcing the exact version, i found that Python3_LIBRARY_INCLUDE_DIR, Python3_INCLUDE_DIR and Python3_NumPy were empty.
Instead Python3_EXECUTABLE was pointing to the python executable, e.g. /usr/bin/python3.7.
Then I tried, using the original YARP CMake's script, the following
$: cmake <YARP_SRC>/bindings/ -DCREATE_PYTHON=true -DPython3_EXECUTABLE=/usr/bin/python3.7and it worked.
traversaro
commented
3 years ago Can you link one of this PPAs and specify with which Ubuntu version you used it?
Actually I have several version of Python but in Arch Linux (using AUR). However, I know that @Arya07 has a similar scenario in Ubuntu. @Arya07 can you maybe specify which
ppadid you add on Ubuntu in order to install them?
In the Arch case, I think the Python3.9 version is always found first regardless of the strategy as, if I understand correctly https://wiki.archlinux.org/index.php/python, /usr/bin/python links to /usr/bin/python3.9.
traversaro
commented
3 years ago After forcing the exact version, i found that
Python3_LIBRARY_INCLUDE_DIR,Python3_INCLUDE_DIRandPython3_NumPywere empty.
This make sense. If you read the docs in https://cmake.org/cmake/help/v3.19/module/FindPython3.html#artifacts-specification and https://cmake.org/cmake/help/v3.19/module/FindPython3.html#result-variables, you can see that Python3_EXECUTABLE is both an input variable used for "Artifacts Specification" and an output variable used to contain the result of the find process, while all the other variables used for "Artifacts Specification" are only used for that. If you want to understand how to set them, check the similarly named Result variables, for example for the expected value of Python3_NumPy_INCLUDE_DIR check the Result Variable Python3_NumPy_INCLUDE_DIRS .
However, if just setting Python3_EXECUTABLE is sufficient, probably we can just suggest that for achieving the workflow that you described.
diegoferigo
commented
3 years ago I am very curious to know if it exists a proper CMake configuration that solves for good this recurring problem. In the past, I never managed to get a proper multiplatform configuration that works with both system interpreters and virtualenvs (or any other kind of user installations).
I typically obtain a proper detection in setups where a user installation needs to be detected (e.g. virtualenv, conda, ...) by specifying Python3_ROOT_DIR (e.g. this action that uses uses this flag). However, this does not work for system interpreters where possibly multiple versions share the same root dir (like @xEnVrE's setup).
This being said, I look forward to the confirmation that passing Python3_EXECUTABLE is enough to solve once and for all the interpreter detection in all major OSs.
It would be great having a simple repo and a Github Workflow that performs this check for all major platforms. The catch would be installing multiple versions of Python from the default package manager (apt, choco, brew) and not actions/setup-python. But I doubt anyone can find enough time to design it properly.
Can you link one of this PPAs and specify with which Ubuntu version you used it?
I'm not sure if this was the case, but in Ubuntu many users use ppa:deadsnakes/ppa for getting non-official Python versions.
xEnVrE
commented
3 years ago Thank you @traversaro for the clarifications on the FindPython variables.
Indeed the output is as follows:
Python3_EXECUTABLE = /usr/bin/python3.7
Python3_LIBRARIES = /usr/lib/libpython3.7m.so
Python3_INCLUDE_DIRS = /usr/include/python3.7m
Python3_NumPy_FOUND = FALSE
Python3_NumPy_INCLUDE_DIRS = However, if just setting Python3_EXECUTABLE is sufficient, probably we can just suggest that for achieving the workflow that you described.
Probably the script uses the executable to obtain all the other informations via sys, but this is just an hypothesis. I agree that, as a starting point, we can suggest to use Python3_EXECUTABLE only.
I'm not sure if this was the case, but in Ubuntu many users use ppa:deadsnakes/ppa for getting non-official Python versions.
Thank you @diegoferigo, I tried with that one on a fresh Ubuntu 18.04 and was able to use an old version of python, 3.5.10, again using Python3_EXECUTABLE=/usr/bin/python3.5 only.
In this case, the variables were populated as:
Python3_EXECUTABLE=/usr/bin/python3.5
Python3_LIBRARIES=/usr/lib/x86_64-linux-gnu/libpython3.5m.so
Python3_INCLUDE_DIRS=/usr/include/python3.5m
Python3_NumPy_FOUND = FALSE
Python3_NumPy_INCLUDE_DIRSI feel that, for now, we can close the issue. Maybe we can decide if it makes sense to add the information on how to force the desired version of Python in the documentation somewhere.
PeterBowman
commented
3 years ago Sorry if not 100% related, but this issue has reminded me of an installation problem which I have just reported in https://github.com/robotology/yarp/issues/2511.
traversaro
commented
3 years ago I don't think that https://github.com/robotology/yarp/issues/2511 is directly related, but thanks for opening it, we had similar issue in robotology-superbuild.
traversaro
commented
3 years ago Maybe we can decide if it makes sense to add the information on how to force the desired version of Python in the documentation somewhere.
At the YARP level, we can probably just add a note in http://www.yarp.it/git-master/yarp_swig.html#yarp_swig_python . As this info is actually useful also at the superbuild level (as long as all the downstream projects use the official FindPython3.cmake) we can also add in https://github.com/robotology/robotology-superbuild/blob/master/doc/profiles.md#python that `cmake -DYCM_EP_ADDITIONAL_CMAKE_ARGS:STRING="-DPython3_EXECUTABLE:BOOL=
xEnVrE
commented
3 years ago If ok for everyone, I could open PRs on both yarp and robotology-superbuild to address what we discussed above.
traversaro
commented
3 years ago If ok for everyone, I could open PRs on both
yarpandrobotology-superbuildto address what we discussed above.
Ok for me!
Arya07
commented
3 years ago Can you link one of this PPAs and specify with which Ubuntu version you used it?
I'm not sure if this was the case, but in Ubuntu many users use
ppa:deadsnakes/ppafor getting non-official Python versions.
Sorry for the late reply. Yes, I confirm that I use ppa:deadsnakes/ppa for getting older versions of Python, when I need. It is quite common.
If ok for everyone, I could open PRs on both yarp and robotology-superbuild to address what we discussed above.
I think it could be very useful.
Is your feature request related to a problem? Please describe.
I would like to be able to compile YARP python bindings with a specific version of Python. At the moment, coherently with the documentation of FindPython, it seems that the line
https://github.com/robotology/yarp/blob/d02f3fd603eeff33609fc88d2a94992622a26b5c/bindings/python/CMakeLists.txt#L13
is taking the most recent version of Python available in the system (or at least in the python root dirs that
FindPythonconsiders by default). For this reason I opened this issue as a feature request, given that python bindings compiles properly.Describe the solution you'd like
I would like to be able, somehow, to specify to the YARP bindings
CMakeLists.txtthe version of Python I need to compile the bindings against.Describe alternatives you've considered
I tried to read the FindPython documentation, in order to understand whether there is a proper way to specify the desired version. It seems that we need to use the optional argument
[version]of thefind_packagemethod.In the following, I will be using
yarp @ v3.4.3.If I modify the line I posted above to
the system detects the user request, but sticks to the most recent version anyways.
E.g. asking
3.7but3.9.1is used.Instead, if I force the exact version using the
EXACTkeyword,I obtain the desired behavior.
Do you think is it possible to add some logic in https://github.com/robotology/yarp/blob/master/bindings/python/CMakeLists.txt, such that the user can decides, if they want, the specific version of python?
Thank you
cc @diegoferigo @Arya07 @traversaro