python-vcstools moved out of incomming for build_common_image.bas

[PeterQFR]

Whilst building the common-image for subt, It seems that python-vcstool has been moved out of incomming.

I was able to get the install to continue by installing it via pip.

so replaced the python-vcstool with python-pip and added a new line in the dockerfile: RUN pip install vcstool

@@ -23,14 +23,14 @@ RUN echo "deb http://packages.ros.org/ros/ubuntu bionic main" > /etc/apt/sources
   && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

 # install ROS (including dependencies for building packages) and other packages
-RUN apt-get -qq update && apt-get -q -y install \
+RUN apt-get  update && apt-get -y install \
   build-essential \
   git \
   python-rosdep \
   python-rosinstall \
+  python-pip \
   python-rosinstall-generator \
   python-wstool \
-  python-vcstool \
   python3-colcon-common-extensions \
   ros-melodic-desktop \
   ros-melodic-teleop-twist-keyboard \
@@ -41,6 +41,7 @@ RUN apt-get -qq update && apt-get -q -y install \
   && rm -rf /var/lib/apt/lists/* \
   && apt-get -qq clean

+RUN pip install vcstool
 # sdformat8-sdf conflicts with sdformat-sdf installed from gazebo
 # so we need to workaround this using a force overwrite
 # Do this before installing ign-gazebo
@@ -101,4 +102,4 @@ RUN [ "/bin/bash" , "-c" , \
 RUN mkdir -p ${SOLUTION_WS_PATH}/src

 # When running a container start in the developer's home folder
-WORKDIR /home/$USERNAME
\ No newline at end of file
+WORKDIR /home/$USERNAME

[adlarkin]

Thank you for looking into this and finding a fix. Another user also reported issues when trying to install python-vcstools with build_common_image.bash.

I just removed my existing images and tried re-running the original script again (installing python-vcstools without pip), and I was able to build the common image (I did not have issues with python-vcstools).

There are a few things I would like to look into before we make any changes to the build_common_image.bash script:

1. Can you show me the original output you get from running build_common_image.bash?
2. Can you try changing line 33 of build_common_image.bash to python-vcstools (notice the s on the end)?
3. What version of Docker are you using?

Edit: can you also remove -qq and -q from line 26 so that we can see the full output from those commands as well?

[pauljurczak]

I experienced build failure related to python-vcstools too:

paul@desktop:~/subt_testbed/subt_hello_world/docker$ ./build_common_image.bash 
Sending build context to Docker daemon  88.62MB
Step 1/23 : FROM nvidia/cudagl:10.1-devel-ubuntu18.04
10.1-devel-ubuntu18.04: Pulling from nvidia/cudagl
7ddbc47eeb70: Already exists 
c1bbdc448b72: Already exists 
8c3b70e39044: Already exists 
45d437916d57: Already exists 
d8f1569ddae6: Already exists 
85386706b020: Already exists 
ee9b457b77d0: Already exists 
be4f3343ecd3: Already exists 
30b4effda4fd: Already exists 
20472fd9be5f: Pull complete 
b0a6a007e7e2: Pull complete 
983a0a481a12: Pull complete 
8a22b42e130d: Pull complete 
c692403f672f: Pull complete 
Digest: sha256:38e418b02b68f1e5ed709ea39d6d1daf9f87f03a3153878ebc9f9e5965fb0c4f
Status: Downloaded newer image for nvidia/cudagl:10.1-devel-ubuntu18.04
 ---> de41e30312cb
Step 2/23 : RUN echo 'Etc/UTC' > /etc/timezone &&   ln -s /usr/share/zoneinfo/Etc/UTC /etc/localtime   && apt-get -qq update && apt-get -q -y install tzdata   && rm -rf /var/lib/apt/lists/*   && apt-get -qq clean
 ---> Running in 3ed62e6bc418
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  tzdata
0 upgraded, 1 newly installed, 0 to remove and 110 not upgraded.
Need to get 190 kB of archives.
After this operation, 3109 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 tzdata all 2020a-0ubuntu0.18.04 [190 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 190 kB in 1s (198 kB/s)
Selecting previously unselected package tzdata.
(Reading database ... 14216 files and directories currently installed.)
Preparing to unpack .../tzdata_2020a-0ubuntu0.18.04_all.deb ...
Unpacking tzdata (2020a-0ubuntu0.18.04) ...
Setting up tzdata (2020a-0ubuntu0.18.04) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline

Current default time zone: 'Etc/UTC'
Local time is now:      Mon Jun 22 19:01:52 UTC 2020.
Universal Time is now:  Mon Jun 22 19:01:52 UTC 2020.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

Removing intermediate container 3ed62e6bc418
 ---> 64c6dca0e11f
Step 3/23 : RUN echo "deb http://packages.ros.org/ros/ubuntu bionic main" > /etc/apt/sources.list.d/ros1-latest.list   && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
 ---> Running in bfca87930538
Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.AXOl47P4fy/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
gpg: key F42ED6FBAB17C654: public key "Open Robotics <info@osrfoundation.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
Removing intermediate container bfca87930538
 ---> 1c8b4ec361de
Step 4/23 : RUN apt-get -qq update && apt-get -q -y install   build-essential   git   python-rosdep   python-rosinstall   python-rosinstall-generator   python-wstool   python-vcstool   python3-colcon-common-extensions   ros-melodic-desktop   ros-melodic-teleop-twist-keyboard   tmux   vim   wget   && rosdep init   && rm -rf /var/lib/apt/lists/*   && apt-get -qq clean
 ---> Running in dec6310542b7
Reading package lists...
Building dependency tree...
Reading state information...
build-essential is already the newest version (12.4ubuntu1).
build-essential set to manually installed.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python-vcstool : Depends: python2 (< 2.8) but it is not installable
                  Depends: python2 (>= 2.7) but it is not installable
                  Depends: python2:any (>= 2.6.6-7~) but it is not installable
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c apt-get -qq update && apt-get -q -y install   build-essential   git   python-rosdep   python-rosinstall   python-rosinstall-generator   python-wstool   python-vcstool   python3-colcon-common-extensions   ros-melodic-desktop   ros-melodic-teleop-twist-keyboard   tmux   vim   wget   && rosdep init   && rm -rf /var/lib/apt/lists/*   && apt-get -qq clean' returned a non-zero code: 100

[adlarkin]

@pauljurczak @PeterQFR according to the vcstool repository, it looks like the package now needs to be python3-vcstool instead of python-vcstool (I too just faced this issue for the first time).

I've created a PR that reflects this change (#8). Would each of you mind trying it out and leaving a comment on the PR stating whether this worked for you or not?

[pauljurczak]

The build was successful, but with a few error and warning messages:

Step 5/23 : RUN echo "deb [trusted=yes] http://packages.osrfoundation.org/gazebo/ubuntu-stable `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-stable.list   && wget http://packages.osrfoundation.org/gazebo.key -O - | apt-key add -   && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654   && apt-get -qq update && apt-get -q -y install   ignition-blueprint   ros-melodic-ros-ign   && rm -rf /var/lib/apt/lists/*   && apt-get -qq clean
 ---> Running in 1a049d8e2b61
--2020-06-23 17:23:09--  http://packages.osrfoundation.org/gazebo.key
Resolving packages.osrfoundation.org (packages.osrfoundation.org)... Warning: apt-key output should not be parsed (stdout is not a terminal)
52.52.171.73
Connecting to packages.osrfoundation.org (packages.osrfoundation.org)|52.52.171.73|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1772 (1.7K) [application/octet-stream]
Saving to: 'STDOUT'

     0K .                                                     100%  193M=0s

2020-06-23 17:23:09 (193 MB/s) - written to stdout [1772/1772]

OK
Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.UGhReobHUz/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

Step 21/23 : RUN [ "/bin/bash" , "-c" ,   "source /opt/ros/melodic/setup.bash   && sudo apt-get -qq update   && rosdep install --from-paths src --ignore-src -r -y   && colcon build   && sudo rm -rf /var/lib/apt/lists/*   && sudo apt-get -qq clean" ]
 ---> Running in 29aaf31fa931
ERROR: the following packages/stacks could not have their rosdep keys resolved
to system dependencies:
subt_ign: Cannot locate rosdep definition for [ignition-transport7]
subt_ros: Cannot locate rosdep definition for [ignition-transport7]
subt_communication_broker: Cannot locate rosdep definition for [ignition-transport7]
subt_communication_model: Cannot locate rosdep definition for [ignition-common3]
subt_rf_interface: Cannot locate rosdep definition for [ignition-math6]
Reading package lists...

Finished <<< subt_ign [26.5s]
--- stderr: subt_ign
/home/developer/workspaces/dependencies_ws/src/subt/subt_ign/src/CommsBrokerPlugin.cc: In member function 'void subt::CommsBrokerPlugin::UpdateIfNewBreadcrumbs()':
/home/developer/workspaces/dependencies_ws/src/subt/subt_ign/src/CommsBrokerPlugin.cc:327:33: warning: unused variable 'name' [-Wunused-variable]
     for (const auto& [name, pose] : this->breadcrumbs)
                                 ^
---
Starting >>> subt_example
Finished <<< subt_example [6.19s]
Starting >>> x3_control
Starting >>> x4_control
Finished <<< x4_control [1.55s]
Finished <<< x3_control [1.57s]
--- stderr: darknet_ros
/home/developer/workspaces/dependencies_ws/src/darknet_ros/darknet/src/gemm.c: In function 'time_gpu':
/home/developer/workspaces/dependencies_ws/src/darknet_ros/darknet/src/gemm.c:232:9: warning: 'cudaThreadSynchronize' is deprecated [-Wdeprecated-declarations]
         cudaThreadSynchronize();
         ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/local/cuda/include/cuda_runtime.h:96:0,
                 from /home/developer/workspaces/dependencies_ws/src/darknet_ros/darknet/include/darknet.h:14,
                 from /home/developer/workspaces/dependencies_ws/src/darknet_ros/darknet/src/utils.h:5,
                 from /home/developer/workspaces/dependencies_ws/src/darknet_ros/darknet/src/gemm.c:2:
/usr/local/cuda/include/cuda_runtime_api.h:957:57: note: declared here
 extern __CUDA_DEPRECATED __host__ cudaError_t CUDARTAPI cudaThreadSynchronize(void);
                                                         ^~~~~~~~~~~~~~~~~~~~~
---
[Processing: darknet_ros]
[Processing: darknet_ros]
[Processing: darknet_ros]
[Processing: darknet_ros]
Finished <<< darknet_ros [2min 44s]

Summary: 40 packages finished [3min 4s]
  2 packages had stderr output: darknet_ros subt_ign
Removing intermediate container 29aaf31fa931
 ---> 8ac02145379d
Step 22/23 : RUN mkdir -p ${SOLUTION_WS_PATH}/src
 ---> Running in ee03b3e32c9a
Removing intermediate container ee03b3e32c9a
 ---> 61721027e533
Step 23/23 : WORKDIR /home/$USERNAME
 ---> Running in 94fff0c7db40
Removing intermediate container 94fff0c7db40
 ---> e8a692e9970a
Successfully built e8a692e9970a
Successfully tagged subt_hello_world_common:latest

[adlarkin]

The build was successful, but with a few error and warning messages

Thanks. Those warnings can be ignored.

The ERROR: the following packages/stacks could not have their rosdep keys resolved to system dependencies message is just saying that the ignition packages don't match any of the keys that rosdep searches for when installing dependencies for the ROS packages in the workspace (one reason why ignition packages are not in the "rosdep keys database" yet is probably because ignition is a relatively new project, but maybe ignition keys will be added later). So, in our case, that's not actually an error since we installed the ignition packages earlier in the Dockerfile.

The warnings you are seeing with darknet/CUDA is because the darknet_ros developers seem to be using an out-of-date API (CUDA should still work with the Docker image, though). This has nothing to do with our Docker setup.

Now that I know #8 fixes this issue, I will merge that PR and close this issue.