rosdep not obeying AMENT_IGNORE and COLCON_IGNORE in a Docker build

russkel commented 3 years ago

I have just found that rosdep only obeys CATKIN_IGNORE files and not AMENT_IGNORE or COLCON_IGNORE.

Should the ignore functionality be expanded to include those as well?

cottsay commented 3 years ago

Can you post your rosdep and catkin_pkg versions? As far as I know, all three should work: https://github.com/ros-infrastructure/catkin_pkg/blob/7ac1ee74e7cd57872b8fe0755cca146c1b345951/src/catkin_pkg/packages.py#L61

russkel commented 3 years ago

I searched through the code and found that as well and thought I must have misunderstood how rosdep was actually working. I am running the latest docker image ros:foxy. My image is building right at the moment and will post those version once it is finished.

russkel commented 3 years ago

root@c296bd12d1f4:/# dpkg-query -l | grep rosdep
ii  python3-rosdep                                  0.19.0-1                             all          rosdep package manager abstraction tool for ROS
ii  python3-rosdep-modules                          0.19.0-1                             all          rosdep package manager abstraction tool for ROS
root@c296bd12d1f4:/# dpkg-query -l | grep catkin
ii  python3-catkin-pkg                              0.4.23-100                           all          catkin package library
ii  python3-catkin-pkg-modules                      0.4.23-1                             all          catkin package library

russkel commented 3 years ago

COLCON_IGNORE works as expected if I try it from the command-line but the same thing was erroring from within the Dockerfile:

#17 21.39 updated cache in /root/.ros/rosdep/sources.cache
#17 21.75 Error(s) in package '/src/ros_templates/robot_description/{{cookiecutter.node_name}}_description/package.xml':
#17 21.75 Package name "{{cookiecutter.node_name}}" does not follow naming conventions
#17 21.75 Package version "{{cookiecutter.version}}" does not follow version conventions
#17 21.75 Invalid email "{{cookiecutter.author_email}}" for person "{{cookiecutter.author_name}}"
#17 ERROR: executor failed running [/bin/sh -c apt-get update && apt-get install -y iproute2 gosu python3-pip less &&   rosdep update && rosdep install --from-path /src --ignore-src -y]: runc did not terminate sucessfully

(There's a COLCON_IGNORE file in /src/ros_templates)

As soon as I add CATKIN_IGNORE the Docker build ignores it.

I think there's another issue at foot here.

nuclearsandwich commented 3 years ago

Can you share a Dockerfile that reproduces the issue?

russkel commented 3 years ago

Can you share a Dockerfile that reproduces the issue?

I can replicate the behavior with a test case Dockerfile:

# syntax=docker/dockerfile:experimental

FROM ros:foxy

# Update rosdeps and install them for the desired packages
RUN --mount=target=/src,source=./src,type=bind,ro \
  ls /src/ros_templates && rosdep update && rosdep install --from-path /src --ignore-src -y

Where ./src is my source tree. However if I copy the ros_templates directory that is giving me issues to a directory on its own and then just mount that directory, it works as expected. Which leads me to believe it's other files in ./src causing this issue.

I thought maybe it was Docker caching the directory and doing something silly with empty files, but the ls clearly shows the COLCON_IGNORE file is there.

nuclearsandwich commented 3 years ago

Without your src directory I can't reproduce the issue with the dockerfile above. If you can share a minimal self-contained dockerfile that shows the issue I can assist with the troubleshooting.

russkel commented 3 years ago

Understood, I can't share my src directory as it is, but I will attempt to get a test case together.

nuclearsandwich commented 3 years ago

I can't share my src directory as it is, but I will attempt to get a test case together.

Thanks. I appreciate the time. As cottsay mentioned, rosdep's ignore handling comes by way of catkin_pkg and has been working (to the best of our knowledge) for some time. So if there's a case where it isn't working as expected we need to figure out whether the behavior or the expectation should change.

130s commented 1 year ago

Just an info: Can't provide reproducible info as my source is private though, I confirmed that rosdep doesn't try to install dependencies from a package COLCON_IGNORE exists in.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal

# apt-cache show python3-rosdep 
:
Version: 0.22.1-1

kmhallen commented 1 year ago

Reproduced here by checking out noetic-devel and humble-devel branches of ros_tutorials at the same time and using CATKIN_IGNORE/COLCON_IGNORE/AMENT_IGNORE to selectively build only one: https://bitbucket.org/DataspeedInc/rosdep_ignore/

ros-infrastructure / rosdep

rosdep not obeying AMENT_IGNORE and COLCON_IGNORE in a Docker build #786