PickNikRobotics / generate_parameter_library

Declarative ROS 2 Parameters
BSD 3-Clause "New" or "Revised" License
224 stars 42 forks source link

ModuleNotFoundError when building with --simlink-install #141

Closed BrunoB81HK closed 10 months ago

BrunoB81HK commented 11 months ago

When building the example_python package inside my workspace, I get the ModuleNotFoundError. This happens everytime I try to build with --symlink-install. I first encountered this bug with my package and I decided to try the example directly to see if it happened with it too and it does. here is the traceback:

Traceback (most recent call last):
  File "/home/user/colcon_ws/install/generate_parameter_module_example/lib/generate_parameter_module_example/test_node", line 33, in <module>
    sys.exit(load_entry_point('generate-parameter-module-example', 'console_scripts', 'test_node')())
  File "/home/user/colcon_ws/install/generate_parameter_module_example/lib/generate_parameter_module_example/test_node", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/colcon_ws/build/generate_parameter_module_example/generate_parameter_module_example/minimal_publisher.py", line 33, in <module>
    from generate_parameter_module_example.admittance_parameters import (
ModuleNotFoundError: No module named 'generate_parameter_module_example.admittance_parameters'

I also checked my PYTHONPATH and I think that it may be related to name collisions.

I saw that @pac48 was referencing to some side-effects when using --symlink-install in #122, is this related?

pac48 commented 11 months ago

@BrunoB81HK Which version of Ubuntu and ROS 2 are you using? I can build with --symlink-install and run the example on this image no problem.

BrunoB81HK commented 11 months ago

I'm using humble with Ubuntu 22.04. We are also using docker and our container is generated from this one.

pac48 commented 11 months ago

Are you building with any extra options? When I run colcon build --symlink-install to build and ros2 run generate_parameter_module_example test_node --ros-args --params-file src/generate_parameter_library/example_python/config/implementation.yaml to run, it works even on the image you mentioned.

BrunoB81HK commented 11 months ago

OK, that is weird. The only other that we use is the following --cmake-args -DCMAKE_BUILD_TYPE=Release. Maybe the error is on my side.

pac48 commented 11 months ago

Okay, that is strange. The only other thing I'd suggest is to ensure that rosdep is run: rosdep install --from-paths src --ignore-src -y before building and running.

BrunoB81HK commented 11 months ago

rosdep install --from-paths src --ignore-src -y is indeed called before building and running.

For my custom package, python can find it, but I think its name collide with another package so that parameters.py is not found. I compiled the differences between a code built with and without --symink-install below.

In my case, using --symink-install cause a duplicated module in /colcon_ws/build that is added in the python path and, since this module is a symlink to the /colcon_ws/src, it doesn't contain the generated parameters.py. The generated parameters.py is theoretically reachable from /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages, but is hidden behind the one in /home/user/colcon_ws/build/pkg_name.

The most surprising thing to me is that it happens to me but not to you. @pac48, do you have any idea why this happens and how I could remedy this problem? Is it a problem with my setup? Or a problem with my module naming?

Installed with --symlink-install

Executed script (from __file__):

/home/user/colcon_ws/build/pkg_name/pkg_name/script.py

Python path (from sys.path):

Paths
1️⃣ /home/user/colcon_ws/install/pkg_name/lib/pkg_name
2️⃣ /home/user/colcon_ws/build/pkg_name
3️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages
/opt/ros/humble/lib/python3.10/site-packages
/opt/ros/humble/local/lib/python3.10/dist-packages
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/usr/local/lib/python3.10/dist-packages
/usr/lib/python3/dist-packages

Reachable modules (from sys.modules):

Module Location
pkg_name.script 2️⃣ /home/user/colcon_ws/build/pkg_name
pkg_name 2️⃣ /home/user/colcon_ws/build/pkg_name

Tree of the directory:

colcon_ws
├── build
│   ├── COLCON_IGNORE
│   └── 2️⃣ pkg_name
│       ├── pkg_name -> /home/user/colcon_ws/src/pkg_name/pkg_name
│       ├── pkg_name.egg-info
│       │   └── ...
│       ├── package.xml -> /home/user/colcon_ws/src/pkg_name/package.xml
│       ├── resource
│       │   └── pkg_name -> /home/user/colcon_ws/src/pkg_name/resource/pkg_name
│       ├── setup.cfg -> /home/user/colcon_ws/src/pkg_name/setup.cfg
│       ├── setup.py -> /home/user/colcon_ws/src/pkg_name/setup.py
│       ├── ...
│       └── share
│           └── pkg_name
│               └── hook
│                   ├── pythonpath_develop.dsv
│                   ├── pythonpath_develop.ps1
│                   └── pythonpath_develop.sh
├── install
│   ├── COLCON_IGNORE
│   ├── pkg_name
│   │   ├── lib
│   │   │   ├── 1️⃣ pkg_name
│   │   │   │   └── entry_point
│   │   │   └── python3.10
│   │   │       └── 3️⃣ site-packages
│   │   │           ├── pkg-name.egg-link
│   │   │           └── pkg_name
│   │   │               └── parameters.py
│   │   └── share
│   │       ├── ...
│   │       └── colcon-core
│   │           └── packages
│   │               └── pkg_name
│   └── ...
└── src
    └── pkg_name
        ├── pkg_name
        │   ├── __init__.py
        │   ├── script.py
        │   └── parameters.yaml
        ├── package.xml
        ├── resource
        │   └── pkg_name
        ├── setup.cfg
        └── setup.py

Installed without --symlink-install

Executed script (from __file__):

/home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages/pkg_name/script.py

Python path (from sys.path):

Paths
1️⃣ /home/user/colcon_ws/install/pkg_name/lib/pkg_name
2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages
/opt/ros/humble/lib/python3.10/site-packages
/opt/ros/humble/local/lib/python3.10/dist-packages
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/usr/local/lib/python3.10/dist-packages
/usr/lib/python3/dist-packages

Reachable modules (from sys.modules):

Module Location
pkg_name.script 2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages
pkg_name 2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages

Tree of the directory:

colcon_ws
├── build
│   ├── COLCON_IGNORE
│   └── pkg_name
│       ├── pkg_name
│       │   └── parameters.py
│       ├── pkg_name.egg-info
│       │   └── ...
│       ├── build
│       │   └── lib
│       │       └── pkg_name
│       │           ├── __init__.py
│       │           └── script.py
│       └── ...
├── install
│   ├── COLCON_IGNORE
│   ├── pkg_name
│   │   ├── lib
│   │   │   ├── 1️⃣ pkg_name
│   │   │   │   └── entry_point
│   │   │   └── python3.10
│   │   │       └── 2️⃣ site-packages
│   │   │           ├── pkg_name-0.1.0-py3.10.egg-info
│   │   │           │   └── ...
│   │   │           └── pkg_name
│   │   │               ├── __init__.py
│   │   │               ├── __pycache__
│   │   │               │   └── ...
│   │   │               ├── script.py
│   │   │               └── parameters.py
│   │   └── share
│   │       ├── ...
│   │       └── colcon-core
│   │           └── packages
│   │               └── pkg_name
│   └── ...
└── src
    └── pkg_name
        ├── pkg_name
        │   ├── __init__.py
        │   ├── script.py
        │   └── parameters.yaml
        ├── package.xml
        ├── resource
        │   └── pkg_name
        ├── setup.cfg
        └── setup.py

Difference in the directory structure

  colcon_ws
  ├── build
  │   ├── COLCON_IGNORE
  │   └── pkg_name
- │       ├── pkg_name
- │       │   └── parameters.py
+ │       ├── pkg_name -> /home/user/colcon_ws/src/pkg_name/pkg_name
  │       ├── pkg_name.egg-info
  │       │   └── ...
- │       ├── build
- │       │   └── lib
- │       │       └── pkg_name
- │       │           ├── __init__.py
- │       │           └── script.py
+ │       ├── package.xml -> /home/user/colcon_ws/src/pkg_name/package.xml
+ │       ├── resource
+ │       │   └── pkg_name -> /home/user/colcon_ws/src/pkg_name/resource/pkg_name
+ │       ├── setup.cfg -> /home/user/colcon_ws/src/pkg_name/setup.cfg
+ │       ├── setup.py -> /home/user/colcon_ws/src/pkg_name/setup.py
+ │       └── share
+ │           └── pkg_name
+ │               └── hook
+ │                   ├── pythonpath_develop.dsv
+ │                   ├── pythonpath_develop.ps1
+ │                   └── pythonpath_develop.sh
  ├── install
  │   ├── COLCON_IGNORE
  │   ├── pkg_name
  │   │   ├── lib
  │   │   │   ├── pkg_name
  │   │   │   │   └── entry_point
  │   │   │   └── python3.10
  │   │   │       └── site-packages
- │   │   │           ├── pkg_name-0.1.0-py3.10.egg-info
- │   │   │           │   └── ...
+ │   │   │           ├── pkg-name.egg-link
  │   │   │           └── pkg_name
- │   │   │               ├── __init__.py
- │   │   │               ├── __pycache__
- │   │   │               │   └── ...
- │   │   │               ├── script.py
  │   │   │               └── parameters.py
  │   │   └── share
  │   │       ├── ...
  │   │       └── colcon-core
  │   │           └── packages
  │   │               └── pkg_name
  │   └── ...
  └── src
      └── pkg_name
          ├── pkg_name
          │   ├── __init__.py
          │   ├── script.py
          │   └── parameters.yaml
          ├── package.xml
          ├── resource
          │   └── pkg_name
          ├── setup.cfg
          └── setup.py
BrunoB81HK commented 11 months ago

@pac48 is there any way you can share me the Dockerfile you used that worked for you?

pac48 commented 11 months ago

@BrunoB81HK this is the image that I used. Its called osrf/ros:humble-desktop-full

BrunoB81HK commented 10 months ago

After much investigating, I found out that the probleme lied in my Dockerfile configuration. The build process was not saving any changes made inside the src/ directory. I'll close this issue since it is not a bug. Thanks a lot for your help!

ahi-bplus commented 3 months ago

@BrunoB81HK can you share the changes you made to fix the problem? I have exactly the same problem with --symlink-install

BrunoB81HK commented 3 months ago

@ahi-bplus Are you building the project inside your Dockerfile? If so, the problem for me was pretty simple.

When using --symlink-install, it creates a new module inside the src/ directory. The module is accessible because, if I remember correctly, the whole python module directory (the one who contains the __init__.py file) is symlinked, not just the files. This is not a problem in itself, but becomes one if you mount your src/ into the container. If you do so, the module inside the install/ directory changes to the one mounted in your src/ directory. For that reason, the parameter module is overwritten.

I can give you some ideas to help with your case:

  1. Avoid using --symlink-install inside the Dockerfile.
  2. Build your project inside the docker container, not the Dockerfile.
  3. If used for development, start using devcontainers.
  4. Find a way to avoid mounting your src/ directory inside the container.

I hope it helps, feel free to ask more questions if you need! :)

ahi-bplus commented 3 months ago

@BrunoB81HK Thanks for your quick reply. I already use a devcontainer for development. I mount my src files accordingly. As soon as I build my ROS node which implements generate_parameter_library using colcon build --symlink-install --merge-install, I get the problem with ModuleNotFoundError. Without --symlink-install --merge-install it works.

ahi-bplus commented 3 months ago

Okay, I have found the problem. It has nothing to do with --symlink-install or --merge-install. The problem comes from my package structure.

package_name/
├── README.md
├── __init__.py
├── config
│   └── default
│       └── default.yaml
├── models
├── package.xml
├── requirements.txt
├── resource
│   └── package_name
├── setup.cfg
├── setup.py
├── src
│   └── package_name
│       ├── __init__.py
│       ├── node.py
│       ├── parameters.yaml

Unfortunately the package_dir={'': 'src'}, of generate_parameter_library is not considered. If I avoid the subdirecory src/package_name everything works. I think this would be worth an issue?

BrunoB81HK commented 3 months ago

If you think this is a bug, you should create a new issue. Even if it's not, that will give others that have the same problem some help too.