Closed traversaro closed 2 years ago
The main problem is that the executable/scripts of ros-galactic-demo-nodes-py
packages get installed in %CONDA_PREFIX%\Library\bin
, meaning that they can be executed directly, i.e.:
demo_talker
works, but they are not found by ros2 run demo_nodes_py <executable>
as this command would searches for the commands in %CONDA_PREFIX%\Library\lib
(see https://github.com/ros2/ros2cli/blob/0.16.1/ros2pkg/ros2pkg/api/__init__.py#L44).
So the problem seems related to the installation process of demo_nodes_py
and why it install the scripts in the wrong location.
Interestingly, on Linux and macOS the scripts are installed in the correct location, i.e. $CONDA_PREFIX/lib/demo_nodes_py
.
Yeah, Windows is unfortunately "special" in that the "unix" prefix is in $CONDA_PREFIX\Library\...
The Library\bin
folder should regularly be on the %PATH% (when activate -> https://github.com/mamba-org/mamba/blob/d4acbf7dc3d4743c805e8b9085033703f823b9d6/libmamba/src/core/activation.cpp#L180-L185) but maybe ROS has a special way of looking for the executables?
Yeah, Windows is unfortunately "special" in that the "unix" prefix is in
$CONDA_PREFIX\Library\...
The problem is not the usual special <install_prefix>
on Windows. The issue is that on Linux/macOS the executables/scripts are installed in <install_prefix>/lib/demo_nodes_py
(correct location), while on Windows they are in <install_prefix>/bin
(wrong location). This is not directly related to what <install_prefix>
is on Windows or *nix. These kind of executables are intentionally meant not be on the path, and to be accessible only via ros2 run
, that looks in <install_prefix>/lib/<pkg_name>
.
By the way, I just tried to build via colcon build the latest commit of demo_nodes_py
package on Windows, and indeed they executables are correctly installed in <install_prefix>/lib/<pkg_name>
.
Hm is there anything wrong in https://github.com/RoboStack/vinca/blob/master/vinca/templates/bld_ament_python.bat.in?
Also, does it just happen for python packages, or also for C++ packages (e.g. demo_nodes_cpp)?
Also, does it just happen for python packages, or also for C++ packages (e.g. demo_nodes_cpp)?
C++ packages are fine, ros2 run demo_nodes_cpp talker
works fine and you can check manually that the executables are installed in <install_prefix>/lib/demo_nodes_cpp
.
Hm is there anything wrong in https://github.com/RoboStack/vinca/blob/master/vinca/templates/bld_ament_python.bat.in?
The --install-scripts=
option indeed seems to be quite related. By the way, the .sh
variant of the same script seems to be quite different: https://github.com/RoboStack/vinca/blob/6777295bc3c140abf0d8ba518b9123050c5c28df/vinca/templates/build_ament_python.sh.in .
Hm we could just try building the demo-nodes-py package with the same pip
command as in the sh .. ?
I rebuilt the demo-nodes-py package using the https://github.com/RoboStack/vinca/commit/23c0dda5298b3e25e68466b6b2732114c0263fac: https://anaconda.org/robostack-experimental/ros-galactic-demo-nodes-py/0.14.3/download/win-64/ros-galactic-demo-nodes-py-0.14.3-py38hae75d3f_1.tar.bz2pip
Does that look better? (by the way, sorry, I overwrote the previous version ..)
Looks like the executables are now in Library\lib :)
I just tested, and I still get:
(ros2ws4) C:\>ros2 run demo_nodes_py demo_talker
No executable found
Looks like the executables are now in Library\lib :)
Ah, I see. The problem is that the executables should be in Library\lib\demo_nodes_py
. Perhaps https://github.com/RoboStack/vinca/pull/28 could help?
Reopening here until we know it's actually fixed
Ops! Now executables are installed in Library\lib\ros-galactic-demo-nodes-py
, but they should be installed Library\lib\demo_nodes_py
.
Looks like it's working now :)
Still leaving it open as it needs a rebuild of probably all windows packages.
I double checked and not indeed it works fine:
(ros2) C:\Users\STraversaro>ros2 run demo_nodes_py demo_talker
[INFO] [1642849699.319292400] [talker]: Publishing: "Hello World: 0"
[INFO] [1642849700.186619500] [talker]: Publishing: "Hello World: 1"
[INFO] [1642849701.185644300] [talker]: Publishing: "Hello World: 2"
Hmm .. now we are running into new issues because of this:
CMake Error at C:/bld/ros-galactic-foonathan-memory-vendor_1643229794134/_h_env/Library/share/ament_cmake_copyright/cmake/ament_copyright.cmake:38 (message):
ament_copyright() could not find program 'ament_copyright'
Which comes from
find_program(ament_copyright_BIN NAMES "ament_copyright")
if(NOT ament_copyright_BIN)
message(FATAL_ERROR "ament_copyright() could not find program 'ament_copyright'")
endif()
So we either have to figure out which package executables should not be installed in lib/package-name
, or fix the cmake to also look in lib/package-name
. I'm not sure whether ament_copyright is going to be the only package which is affected - I am guessing ament_linter and whatever else is out there might also be affected.
Any ideas @wolfv @traversaro?
Maybe we can also ping @ooeygui and @nuclearsandwich ... maybe they know how to identify the packages where the executable should end up in lib
vs. bin
?
Hmm, I am wondering, what happens if we don't give that command line argument at all?
We can also check what colcon or bloom are doing, at least in the demo_nodes_py
it was doing the right thing with no additional options.
Yeah, I was just digging more into it ... some packages have a setup.cfg file:
https://github.com/ros2/demos/blob/master/demo_nodes_py/setup.cfg
That seems to give an argument for the install_script
argument that we pass manually. It's also found in this test package: https://github.com/colcon/colcon-bundle/blob/84dd04d238fa2699b7c804e51733bad4b9aed29e/integration/ros2_workspace/src/test_py_package/setup.cfg
In ament_lint we don't find that setup.cfg file: https://github.com/ament/ament_lint/tree/master/ament_copyright
I didn't find anything in the ms-iot pipelines and I also didn't find anything more in colcon so far. They do some setup.py introspection but nothing that changes the --install-scripts
arguments as far as I can see (https://github.com/colcon/colcon-python-setup-py/tree/master/colcon_python_setup_py)
So maybe not setting anything is the solution?
Maybe we can also ping @ooeygui and @nuclearsandwich ... maybe they know how to identify the packages where the executable should end up in
lib
vs.bin
?
In order to get the scripts into the /opt/ros/bin directory our bloom templates usually override the install_scripts dir for python packages. However when a package is designed to use the ros2run/libexec pattern they use the same mechanism to set the install_scripts to the libexec dir in setup.cfg. Bloom is configured to look for this:
and only override the setting via pybuild if there's none configured in setup.cfg. https://github.com/ros-infrastructure/bloom/blob/ab8ebb8ce2503801054f4a34137a3aec3a843ad7/bloom/generators/debian/templates/ament_python/rules.em#L24
Ah, interesting, thanks for this @nuclearsandwich! The remaining code of interest: https://github.com/ros-infrastructure/bloom/blob/53beb5284fbe7b453af7cad3ef62d159168862bf/bloom/generators/debian/generator.py#L395-L406
So we need to check whether a setup.cfg exists and an install option is specified. A bit trickier with vinca, as we neither have access to the source files at this stage, and use a bash script instead of python. Any ideas @wolfv?
I guess we need to get our bash-foo going :)
On Linux / macOS we could try something like
INSTALL_SCRIPTS_ARG=--install-scripts=$PREFIX/lib
if grep -q "install[-_]scripts" setup.cfg; then
INSTALL_SCRIPTS_ARG=""
fi
echo "ARG: $INSTALL_SCRIPTS_ARG"
On Windows I think we can use findstr
. Going to figure that out now :)
On Windows, this should work:
set "INSTALL_SCRIPTS_ARG=--install-scripts=%_LIBRARY_PREFIX_PATH%\lib"
findstr install[-_]scripts setup.cfg
if %errorlevel% == 0 (
set INSTALL_SCRIPTS_ARG=""
)
echo %INSTALL_SCRIPTS_ARG%
Needs https://github.com/RoboStack/vinca/pull/29 & #75
Hopefully fixed with the latest rebuild, to be confirmed though
demo-nodes-py still needs to be rebuilt:
mamba create -n ros_galactic ros-galactic-demo-nodes-py python=3.9 -c robostack-experimental -c conda-forge --no-channel-priority --override-channels
Encountered problems while solving:
- package ros-galactic-demo-nodes-py-0.14.3-py38hae75d3f_0 requires python_abi 3.8.*, but none of the providers can <..>
https://anaconda.org/robostack-experimental/ros-galactic-demo-nodes-py/files
The executable is now correctly found, but another error emerged:
ros2 run demo_nodes_py demo_talker
(ros2) C:\Users\STraversaro>ros2 run demo_nodes_py demo_talker
Traceback (most recent call last):
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Library\lib\demo_nodes_py\demo_talker-script.py", line 33, in <module>
sys.exit(load_entry_point('demo-nodes-py==0.14.3', 'console_scripts', 'demo_talker')())
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Library\lib\demo_nodes_py\demo_talker-script.py", line 25, in importlib_load_entry_point
return next(matches).load()
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\lib\importlib\metadata.py", line 86, in load
module = import_module(match.group('module'))
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\demo_nodes_py\topics\talker.py", line 16, in <module>
from rclpy.node import Node
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rclpy\node.py", line 43, in <module>
from rclpy.client import Client
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rclpy\client.py", line 22, in <module>
from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rclpy\impl\implementation_singleton.py", line 32, in <module>
rclpy_implementation = import_c_library('._rclpy_pybind11', package)
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rpyutils\import_c_library.py", line 39, in import_c_library
return importlib.import_module(name, package=package)
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension 'C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\_rclpy_pybind11.cp39-win_amd64.pyd' isn't present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-without-library-present-on-the-system' for possible solutions
Environment created with:
mamba create -n ros2 -c conda-forge -c robostack-experimental ros-galactic-desktop
The problem can be reproduced more easily without any use of ros2 run
:
(ros2) C:\Users\STraversaro>python -c "from rclpy.node import Node"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rclpy\node.py", line 43, in <module>
from rclpy.client import Client
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rclpy\client.py", line 22, in <module>
from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rclpy\impl\implementation_singleton.py", line 32, in <module>
rclpy_implementation = import_c_library('._rclpy_pybind11', package)
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\rpyutils\import_c_library.py", line 39, in import_c_library
return importlib.import_module(name, package=package)
File "C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension 'C:\Users\STraversaro\AppData\Local\mambaforge\envs\ros2\Lib\site-packages\_rclpy_pybind11.cp39-win_amd64.pyd' isn't present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-without-library-present-on-the-system' for possible solutions
So it could be considered a different issue.
With the workaround in https://github.com/RoboStack/ros-galactic/issues/96#issuecomment-1086014185, the node finally starts fine:
(ros2) C:\Users\STraversaro>ros2 run demo_nodes_py demo_talker
[INFO] [1648825676.436909000] [talker]: Publishing: "Hello World: 0"
[INFO] [1648825677.348159400] [talker]: Publishing: "Hello World: 1"
[INFO] [1648825678.347715300] [talker]: Publishing: "Hello World: 2"
[INFO] [1648825679.359195500] [talker]: Publishing: "Hello World: 3"
and so on so forth
See #96 instead as this issue is resolved
Finally working fine after fixing https://github.com/RoboStack/ros-galactic/issues/96 :
(ros2) C:\Users\STraversaro>ros2 run demo_nodes_py demo_talker
[INFO] [1648985579.630156000] [talker]: Publishing: "Hello World: 0"
[INFO] [1648985580.556038200] [talker]: Publishing: "Hello World: 1"
[INFO] [1648985581.542916100] [talker]: Publishing: "Hello World: 2"
[INFO] [1648985582.546812300] [talker]: Publishing: "Hello World: 3"
On my Windows system, if I try to run:
it fails with:
Other similar nodes instead work fine, for example:
whose output is:
Environment (
conda list
):Details about
conda
and system (conda info
):