ros-tooling / cross_compile

A tool to build ROS and ROS2 workspaces for various targets
Apache License 2.0
188 stars 59 forks source link

Cannot run in gitlab-ci with docker executor: echo 'No src/ directory found at /ws, did you remember to mount your workspace?' #313

Closed kjeremy closed 3 years ago

kjeremy commented 3 years ago

Description

I am attempting to run this script to cross-compile for melodic. At the moment it fails accessing the workspace.

Expected Behavior

It should succeed.

Actual Behavior

It fails:

$ ros_cross_compile ros_ws --arch aarch64 --os ubuntu --rosdistro melodic --custom-setup-script ./setup.sh  --skip-rosdep-keys delphi_esr
INFO:Rosdep Gatherer:Building rosdep collector image: ros_cross_compile:rosdep
INFO:Docker Client:Step 1/10 : FROM ubuntu:bionic
INFO:Docker Client: ---> 81bcf752ac3d
INFO:Docker Client:Step 2/10 : RUN apt-get update && apt-get install --no-install-recommends -y       dirmngr       gnupg2     && rm -rf /var/lib/apt/lists/*
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 66e71fc01496
INFO:Docker Client:Step 3/10 : RUN apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 56b494134900
INFO:Docker Client:Step 4/10 : RUN echo "deb http://packages.ros.org/ros/ubuntu bionic main" > /etc/apt/sources.list.d/ros-latest.list
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 8fa12f0d39a2
INFO:Docker Client:Step 5/10 : RUN apt-get update && apt-get install --no-install-recommends -y       python-rosdep       python3-colcon-common-extensions     && rm -rf /var/lib/apt/lists/*
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 4b3a4f918702
INFO:Docker Client:Step 6/10 : RUN rosdep init
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> d760a8a8591e
INFO:Docker Client:Step 7/10 : COPY gather_rosdeps.sh /root/
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 77fee945736d
INFO:Docker Client:Step 8/10 : RUN mkdir -p /ws
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> d7c12a2077c3
INFO:Docker Client:Step 9/10 : WORKDIR /ws
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> e5eba9dc0d20
INFO:Docker Client:Step 10/10 : ENTRYPOINT ["/root/gather_rosdeps.sh"]
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 217a5be9d9fe
INFO:Docker Client:Successfully built 217a5be9d9fe
INFO:Docker Client:Successfully tagged ros_cross_compile:rosdep
INFO:Rosdep Gatherer:Running rosdep collector image on workspace /builds/ros_ws
INFO:Docker Client:+ '[' '!' -d ./src ']'
INFO:Docker Client:++ pwd
INFO:Docker Client:+ echo 'No src/ directory found at /ws, did you remember to mount your workspace?'
INFO:Docker Client:+ exit 1
INFO:Docker Client:No src/ directory found at /ws, did you remember to mount your workspace?
Traceback (most recent call last):
  File "/usr/local/bin/ros_cross_compile", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/ros_cross_compile.py", line 227, in main
    cross_compile_pipeline(args, data_collector, platform)
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/ros_cross_compile.py", line 214, in cross_compile_pipeline
    stage(platform, docker_client, ros_workspace_dir, options, data_collector)
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/dependencies.py", line 136, in __call__
    custom_data_dir=options.custom_data_dir)
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/dependencies.py", line 87, in gather_rosdeps
    volumes=volumes,
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/docker_client.py", line 145, in run_container
    image_name, exit_code, '', image_name, 'See above ^')
docker.errors.ContainerError: Command '' in image 'ros_cross_compile:rosdep' returned non-zero exit status 1: See above ^

To Reproduce

** Steps to reproduce the behavior, e.g. Create a gitlab runner with /var/run/docker.sock mounted. In the gitlab.yaml:

image: ros:melodic-ros-core-bionic

variables:
  # TODO:  Not concurrency safe: https://docs.gitlab.com/ee/ci/runners/README.html#handling-concurrency
  GIT_CLONE_PATH: $CI_BUILDS_DIR/ros_ws/src/$CI_PROJECT_NAME

build-stuff:
  stage: build
  tags:
    - ros
  before_script:
    - cd $CI_BUILDS_DIR
    - apt update
    - apt install -y qemu-user-static python3-pip tree
    - pip3 install ros_cross_compile
    - |
      cat > setup.sh << EOF
      #!/bin/sh
      echo "deb [trusted=yes] https://s3.amazonaws.com/autonomoustuff-repo/ bionic main" > /etc/apt/sources.list.d/autonomousstuff.list
      apt update
      apt install -y ros-melodic-delphi-esr
      EOF
    - |
      cat > ros_ws/defaults.yaml << EOF
      list:
        packages-up-to: [robot_bringup]
      build:
        packages-up-to: [robot_bringup]
      EOF
  script:
    - pwd
    - tree -L 3
    - ros_cross_compile ros_ws --arch aarch64 --os ubuntu --rosdistro melodic --custom-setup-script ./setup.sh  --skip-rosdep-keys delphi_esr

The build output:

Running with gitlab-runner 13.11.0 (7f7a4bb0)
  on Shared ROS runner M7ymcZ_m
Preparing the "docker" executor 00:02
Using Docker executor with image ros:melodic-ros-core-bionic ...
Pulling docker image ros:melodic-ros-core-bionic ...
Using docker image sha256:edad2de99c6a4f745980c0c51a241d9a200d421170b496f602a5785e9b29bc8e for ros:melodic-ros-core-bionic with digest ros@sha256:6a4f62ca13b7d2db437fe06471e7b053fbfd80c5639bb93c903857c93e2a2b6a ...
Preparing environment 00:00
Running on runner-m7ymczm-project-275-concurrent-0 via a56f0af87da2...
Getting source from Git repository 00:02
Fetching changes...
Reinitialized existing Git repository in /builds/ros_ws/src/robot/.git/
Checking out 87a4d662 as cross-compile...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:40
Using docker image sha256:edad2de99c6a4f745980c0c51a241d9a200d421170b496f602a5785e9b29bc8e for ros:melodic-ros-core-bionic with digest ros@sha256:6a4f62ca13b7d2db437fe06471e7b053fbfd80c5639bb93c903857c93e2a2b6a ...
$ cd $CI_BUILDS_DIR
$ apt update
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://packages.ros.org/ros/ubuntu bionic InRelease [4680 B]
Get:3 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1412 kB]
Get:7 http://packages.ros.org/ros/ubuntu bionic/main amd64 Packages [756 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2152 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [423 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [24.7 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2583 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [452 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.6 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2183 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Fetched 23.4 MB in 11s (2066 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
All packages are up to date.
$ apt install -y qemu-user-static python3-pip tree
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  binfmt-support build-essential cpp cpp-7 dbus g++ g++-7 gcc gcc-7
  libapparmor1 libasan4 libcc1-0 libcilkrts5 libdbus-1-3 libgcc-7-dev libisl19
  liblsan0 libmpc3 libmpfr6 libmpx2 libpipeline1 libstdc++-7-dev libubsan0
  python-pip-whl python3-asn1crypto python3-cffi-backend python3-crypto
  python3-cryptography python3-dbus python3-gi python3-idna python3-keyring
  python3-keyrings.alt python3-pkg-resources python3-secretstorage
  python3-setuptools python3-six python3-wheel python3-xdg
Suggested packages:
  cpp-doc gcc-7-locales default-dbus-session-bus | dbus-session-bus
  g++-multilib g++-7-multilib gcc-7-doc libstdc++6-7-dbg gcc-multilib
  manpages-dev autoconf automake libtool flex bison gdb gcc-doc gcc-7-multilib
  libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan4-dbg
  liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libmpx2-dbg
  libquadmath0-dbg libstdc++-7-doc python-crypto-doc python-cryptography-doc
  python3-cryptography-vectors python-dbus-doc python3-dbus-dbg gnome-keyring
  libkf5wallet-bin gir1.2-gnomekeyring-1.0 python-secretstorage-doc
  python-setuptools-doc
The following NEW packages will be installed:
  binfmt-support build-essential cpp cpp-7 dbus g++ g++-7 gcc gcc-7
  libapparmor1 libasan4 libcc1-0 libcilkrts5 libdbus-1-3 libgcc-7-dev libisl19
  liblsan0 libmpc3 libmpfr6 libmpx2 libpipeline1 libstdc++-7-dev libubsan0
  python-pip-whl python3-asn1crypto python3-cffi-backend python3-crypto
  python3-cryptography python3-dbus python3-gi python3-idna python3-keyring
  python3-keyrings.alt python3-pip python3-pkg-resources python3-secretstorage
  python3-setuptools python3-six python3-wheel python3-xdg qemu-user-static
  tree
0 upgraded, 42 newly installed, 0 to remove and 0 not upgraded.
Need to get 46.7 MB of archives.
After this operation, 227 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapparmor1 amd64 2.12-4ubuntu5.1 [31.3 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libdbus-1-3 amd64 1.12.2-1ubuntu1.2 [175 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 dbus amd64 1.12.2-1ubuntu1.2 [150 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dbus amd64 1.2.6-1 [89.9 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-gi amd64 3.26.1-2ubuntu1 [153 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpipeline1 amd64 1.5.0-1 [25.3 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 binfmt-support amd64 2.1.8-2 [51.6 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libisl19 amd64 0.19-1 [551 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmpfr6 amd64 4.0.1-1 [243 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libmpc3 amd64 1.1.0-1 [40.8 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 cpp-7 amd64 7.5.0-3ubuntu1~18.04 [8591 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 cpp amd64 4:7.4.0-1ubuntu2.3 [27.7 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcc1-0 amd64 8.4.0-1ubuntu1~18.04 [39.4 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libasan4 amd64 7.5.0-3ubuntu1~18.04 [358 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 liblsan0 amd64 8.4.0-1ubuntu1~18.04 [133 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libubsan0 amd64 7.5.0-3ubuntu1~18.04 [126 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcilkrts5 amd64 7.5.0-3ubuntu1~18.04 [42.5 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmpx2 amd64 8.4.0-1ubuntu1~18.04 [11.6 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libgcc-7-dev amd64 7.5.0-3ubuntu1~18.04 [2378 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 gcc-7 amd64 7.5.0-3ubuntu1~18.04 [9381 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 gcc amd64 4:7.4.0-1ubuntu2.3 [5184 B]
Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libstdc++-7-dev amd64 7.5.0-3ubuntu1~18.04 [1471 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 g++-7 amd64 7.5.0-3ubuntu1~18.04 [9697 kB]
Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 g++ amd64 4:7.4.0-1ubuntu2.3 [1568 B]
Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 build-essential amd64 12.4ubuntu1 [4758 B]
Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python-pip-whl all 9.0.1-2.3~ubuntu1.18.04.4 [1653 kB]
Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB]
Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB]
Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-crypto amd64 2.6.1-8ubuntu2 [244 kB]
Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-idna all 2.6-1 [32.5 kB]
Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-six all 1.11.0-2 [11.4 kB]
Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB]
Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-secretstorage all 2.3.1-2 [12.1 kB]
Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-keyring all 10.6.0-1 [26.7 kB]
Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-keyrings.alt all 3.0-1 [16.6 kB]
Get:36 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-pip all 9.0.1-2.3~ubuntu1.18.04.4 [114 kB]
Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pkg-resources all 39.0.1-2 [98.8 kB]
Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB]
Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-wheel all 0.30.0-0.2 [36.5 kB]
Get:40 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-xdg all 0.25-4ubuntu1.1 [31.3 kB]
Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 qemu-user-static amd64 1:2.11+dfsg-1ubuntu7.36 [9989 kB]
Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 tree amd64 1.7.0-5 [40.7 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 46.7 MB in 9s (5249 kB/s)
Selecting previously unselected package libapparmor1:amd64.
(Reading database ... 41897 files and directories currently installed.)
Preparing to unpack .../00-libapparmor1_2.12-4ubuntu5.1_amd64.deb ...
Unpacking libapparmor1:amd64 (2.12-4ubuntu5.1) ...
Selecting previously unselected package libdbus-1-3:amd64.
Preparing to unpack .../01-libdbus-1-3_1.12.2-1ubuntu1.2_amd64.deb ...
Unpacking libdbus-1-3:amd64 (1.12.2-1ubuntu1.2) ...
Selecting previously unselected package dbus.
Preparing to unpack .../02-dbus_1.12.2-1ubuntu1.2_amd64.deb ...
Unpacking dbus (1.12.2-1ubuntu1.2) ...
Selecting previously unselected package python3-dbus.
Preparing to unpack .../03-python3-dbus_1.2.6-1_amd64.deb ...
Unpacking python3-dbus (1.2.6-1) ...
Selecting previously unselected package python3-gi.
Preparing to unpack .../04-python3-gi_3.26.1-2ubuntu1_amd64.deb ...
Unpacking python3-gi (3.26.1-2ubuntu1) ...
Selecting previously unselected package libpipeline1:amd64.
Preparing to unpack .../05-libpipeline1_1.5.0-1_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.0-1) ...
Selecting previously unselected package binfmt-support.
Preparing to unpack .../06-binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package libisl19:amd64.
Preparing to unpack .../07-libisl19_0.19-1_amd64.deb ...
Unpacking libisl19:amd64 (0.19-1) ...
Selecting previously unselected package libmpfr6:amd64.
Preparing to unpack .../08-libmpfr6_4.0.1-1_amd64.deb ...
Unpacking libmpfr6:amd64 (4.0.1-1) ...
Selecting previously unselected package libmpc3:amd64.
Preparing to unpack .../09-libmpc3_1.1.0-1_amd64.deb ...
Unpacking libmpc3:amd64 (1.1.0-1) ...
Selecting previously unselected package cpp-7.
Preparing to unpack .../10-cpp-7_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking cpp-7 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package cpp.
Preparing to unpack .../11-cpp_4%3a7.4.0-1ubuntu2.3_amd64.deb ...
Unpacking cpp (4:7.4.0-1ubuntu2.3) ...
Selecting previously unselected package libcc1-0:amd64.
Preparing to unpack .../12-libcc1-0_8.4.0-1ubuntu1~18.04_amd64.deb ...
Unpacking libcc1-0:amd64 (8.4.0-1ubuntu1~18.04) ...
Selecting previously unselected package libasan4:amd64.
Preparing to unpack .../13-libasan4_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking libasan4:amd64 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package liblsan0:amd64.
Preparing to unpack .../14-liblsan0_8.4.0-1ubuntu1~18.04_amd64.deb ...
Unpacking liblsan0:amd64 (8.4.0-1ubuntu1~18.04) ...
Selecting previously unselected package libubsan0:amd64.
Preparing to unpack .../15-libubsan0_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking libubsan0:amd64 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package libcilkrts5:amd64.
Preparing to unpack .../16-libcilkrts5_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking libcilkrts5:amd64 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package libmpx2:amd64.
Preparing to unpack .../17-libmpx2_8.4.0-1ubuntu1~18.04_amd64.deb ...
Unpacking libmpx2:amd64 (8.4.0-1ubuntu1~18.04) ...
Selecting previously unselected package libgcc-7-dev:amd64.
Preparing to unpack .../18-libgcc-7-dev_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking libgcc-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package gcc-7.
Preparing to unpack .../19-gcc-7_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking gcc-7 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package gcc.
Preparing to unpack .../20-gcc_4%3a7.4.0-1ubuntu2.3_amd64.deb ...
Unpacking gcc (4:7.4.0-1ubuntu2.3) ...
Selecting previously unselected package libstdc++-7-dev:amd64.
Preparing to unpack .../21-libstdc++-7-dev_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking libstdc++-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package g++-7.
Preparing to unpack .../22-g++-7_7.5.0-3ubuntu1~18.04_amd64.deb ...
Unpacking g++-7 (7.5.0-3ubuntu1~18.04) ...
Selecting previously unselected package g++.
Preparing to unpack .../23-g++_4%3a7.4.0-1ubuntu2.3_amd64.deb ...
Unpacking g++ (4:7.4.0-1ubuntu2.3) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../24-build-essential_12.4ubuntu1_amd64.deb ...
Unpacking build-essential (12.4ubuntu1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../25-python-pip-whl_9.0.1-2.3~ubuntu1.18.04.4_all.deb ...
Unpacking python-pip-whl (9.0.1-2.3~ubuntu1.18.04.4) ...
Selecting previously unselected package python3-asn1crypto.
Preparing to unpack .../26-python3-asn1crypto_0.24.0-1_all.deb ...
Unpacking python3-asn1crypto (0.24.0-1) ...
Selecting previously unselected package python3-cffi-backend.
Preparing to unpack .../27-python3-cffi-backend_1.11.5-1_amd64.deb ...
Unpacking python3-cffi-backend (1.11.5-1) ...
Selecting previously unselected package python3-crypto.
Preparing to unpack .../28-python3-crypto_2.6.1-8ubuntu2_amd64.deb ...
Unpacking python3-crypto (2.6.1-8ubuntu2) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../29-python3-idna_2.6-1_all.deb ...
Unpacking python3-idna (2.6-1) ...
Selecting previously unselected package python3-six.
Preparing to unpack .../30-python3-six_1.11.0-2_all.deb ...
Unpacking python3-six (1.11.0-2) ...
Selecting previously unselected package python3-cryptography.
Preparing to unpack .../31-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ...
Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ...
Selecting previously unselected package python3-secretstorage.
Preparing to unpack .../32-python3-secretstorage_2.3.1-2_all.deb ...
Unpacking python3-secretstorage (2.3.1-2) ...
Selecting previously unselected package python3-keyring.
Preparing to unpack .../33-python3-keyring_10.6.0-1_all.deb ...
Unpacking python3-keyring (10.6.0-1) ...
Selecting previously unselected package python3-keyrings.alt.
Preparing to unpack .../34-python3-keyrings.alt_3.0-1_all.deb ...
Unpacking python3-keyrings.alt (3.0-1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../35-python3-pip_9.0.1-2.3~ubuntu1.18.04.4_all.deb ...
Unpacking python3-pip (9.0.1-2.3~ubuntu1.18.04.4) ...
Selecting previously unselected package python3-pkg-resources.
Preparing to unpack .../36-python3-pkg-resources_39.0.1-2_all.deb ...
Unpacking python3-pkg-resources (39.0.1-2) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../37-python3-setuptools_39.0.1-2_all.deb ...
Unpacking python3-setuptools (39.0.1-2) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../38-python3-wheel_0.30.0-0.2_all.deb ...
Unpacking python3-wheel (0.30.0-0.2) ...
Selecting previously unselected package python3-xdg.
Preparing to unpack .../39-python3-xdg_0.25-4ubuntu1.1_all.deb ...
Unpacking python3-xdg (0.25-4ubuntu1.1) ...
Selecting previously unselected package qemu-user-static.
Preparing to unpack .../40-qemu-user-static_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...
Unpacking qemu-user-static (1:2.11+dfsg-1ubuntu7.36) ...
Selecting previously unselected package tree.
Preparing to unpack .../41-tree_1.7.0-5_amd64.deb ...
Unpacking tree (1.7.0-5) ...
Setting up python-pip-whl (9.0.1-2.3~ubuntu1.18.04.4) ...
Setting up tree (1.7.0-5) ...
Setting up libcc1-0:amd64 (8.4.0-1ubuntu1~18.04) ...
Setting up python3-cffi-backend (1.11.5-1) ...
Setting up libasan4:amd64 (7.5.0-3ubuntu1~18.04) ...
Setting up libcilkrts5:amd64 (7.5.0-3ubuntu1~18.04) ...
Setting up python3-crypto (2.6.1-8ubuntu2) ...
Setting up libubsan0:amd64 (7.5.0-3ubuntu1~18.04) ...
Setting up python3-idna (2.6-1) ...
Setting up python3-xdg (0.25-4ubuntu1.1) ...
Setting up python3-six (1.11.0-2) ...
Setting up libmpfr6:amd64 (4.0.1-1) ...
Setting up libpipeline1:amd64 (1.5.0-1) ...
Setting up python3-wheel (0.30.0-0.2) ...
Setting up python3-pkg-resources (39.0.1-2) ...
Setting up python3-gi (3.26.1-2ubuntu1) ...
Setting up liblsan0:amd64 (8.4.0-1ubuntu1~18.04) ...
Setting up python3-asn1crypto (0.24.0-1) ...
Setting up libmpx2:amd64 (8.4.0-1ubuntu1~18.04) ...
Setting up libapparmor1:amd64 (2.12-4ubuntu5.1) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1.18.04.4) ...
Setting up qemu-user-static (1:2.11+dfsg-1ubuntu7.36) ...
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
Setting up libmpc3:amd64 (1.1.0-1) ...
Setting up python3-setuptools (39.0.1-2) ...
Setting up libdbus-1-3:amd64 (1.12.2-1ubuntu1.2) ...
Setting up libisl19:amd64 (0.19-1) ...
Setting up python3-cryptography (2.1.4-1ubuntu1.4) ...
Setting up binfmt-support (2.1.8-2) ...
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
mount: /proc/sys/fs/binfmt_misc: permission denied.
update-binfmts: warning: Couldn't mount the binfmt_misc filesystem on /proc/sys/fs/binfmt_misc.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up python3-dbus (1.2.6-1) ...
Setting up python3-keyrings.alt (3.0-1) ...
Setting up libgcc-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ...
Setting up cpp-7 (7.5.0-3ubuntu1~18.04) ...
Setting up libstdc++-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ...
Setting up dbus (1.12.2-1ubuntu1.2) ...
Setting up python3-secretstorage (2.3.1-2) ...
Setting up cpp (4:7.4.0-1ubuntu2.3) ...
Setting up python3-keyring (10.6.0-1) ...
Setting up gcc-7 (7.5.0-3ubuntu1~18.04) ...
Setting up g++-7 (7.5.0-3ubuntu1~18.04) ...
Setting up gcc (4:7.4.0-1ubuntu2.3) ...
Setting up g++ (4:7.4.0-1ubuntu2.3) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/c++.1.gz because associated file /usr/share/man/man1/g++.1.gz (of link group c++) doesn't exist
Setting up build-essential (12.4ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
$ pip3 install ros_cross_compile
Collecting ros_cross_compile
  Downloading https://files.pythonhosted.org/packages/d1/5d/d05a039c06d95ebc8b34c86a08f5599d5fcdb0819358b009ee2bb8689158/ros_cross_compile-0.6.0-py3-none-any.whl
Collecting docker<3,>=2 (from ros_cross_compile)
  Downloading https://files.pythonhosted.org/packages/6b/01/74d61c7de9fb99cebbbfab62ee5c88a9b23e8b1c25a57eded5fc22e87bd2/docker-2.7.0-py2.py3-none-any.whl (119kB)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from ros_cross_compile)
Collecting requests!=2.18.0,>=2.14.2 (from docker<3,>=2->ros_cross_compile)
  Downloading https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl (61kB)
Collecting docker-pycreds>=0.2.1 (from docker<3,>=2->ros_cross_compile)
  Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl
Collecting websocket-client>=0.32.0 (from docker<3,>=2->ros_cross_compile)
  Downloading https://files.pythonhosted.org/packages/ba/d1/501076b54481412df1bc4cdd1fe479f66e17857c63ec5981bedcdc2ca793/websocket_client-1.0.0-py2.py3-none-any.whl (68kB)
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from docker<3,>=2->ros_cross_compile)
Collecting certifi>=2017.4.17 (from requests!=2.18.0,>=2.14.2->docker<3,>=2->ros_cross_compile)
  Downloading https://files.pythonhosted.org/packages/5e/a0/5f06e1e1d463903cf0c0eebeb751791119ed7a4b3737fdc9a77f1cdfb51f/certifi-2020.12.5-py2.py3-none-any.whl (147kB)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests!=2.18.0,>=2.14.2->docker<3,>=2->ros_cross_compile)
Collecting chardet<5,>=3.0.2 (from requests!=2.18.0,>=2.14.2->docker<3,>=2->ros_cross_compile)
  Downloading https://files.pythonhosted.org/packages/19/c7/fa589626997dd07bd87d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl (178kB)
Collecting urllib3<1.27,>=1.21.1 (from requests!=2.18.0,>=2.14.2->docker<3,>=2->ros_cross_compile)
  Downloading https://files.pythonhosted.org/packages/09/c6/d3e3abe5b4f4f16cf0dfc9240ab7ce10c2baa0e268989a4e3ec19e90c84e/urllib3-1.26.4-py2.py3-none-any.whl (153kB)
Installing collected packages: certifi, chardet, urllib3, requests, docker-pycreds, websocket-client, docker, ros-cross-compile
Successfully installed certifi-2020.12.5 chardet-4.0.0 docker-2.7.0 docker-pycreds-0.4.0 requests-2.25.1 ros-cross-compile-0.6.0 urllib3-1.26.4 websocket-client-1.0.0
$ cat > setup.sh << EOF # collapsed multi-line command
$ cat > ros_ws/defaults.yaml << EOF # collapsed multi-line command
$ pwd
/builds
$ tree -L 3
.
├── ros_ws
│   ├── cc_internals
│   │   ├── aarch64-ubuntu-melodic
│   │   └── metrics
│   ├── defaults.yaml
│   └── src
│       ├── robot
│       └── robot.tmp
└── setup.sh
7 directories, 2 files
$ ros_cross_compile ros_ws --arch aarch64 --os ubuntu --rosdistro melodic --custom-setup-script ./setup.sh  --skip-rosdep-keys delphi_esr
INFO:Rosdep Gatherer:Building rosdep collector image: ros_cross_compile:rosdep
INFO:Docker Client:Step 1/10 : FROM ubuntu:bionic
INFO:Docker Client: ---> 81bcf752ac3d
INFO:Docker Client:Step 2/10 : RUN apt-get update && apt-get install --no-install-recommends -y       dirmngr       gnupg2     && rm -rf /var/lib/apt/lists/*
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 66e71fc01496
INFO:Docker Client:Step 3/10 : RUN apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 56b494134900
INFO:Docker Client:Step 4/10 : RUN echo "deb http://packages.ros.org/ros/ubuntu bionic main" > /etc/apt/sources.list.d/ros-latest.list
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 8fa12f0d39a2
INFO:Docker Client:Step 5/10 : RUN apt-get update && apt-get install --no-install-recommends -y       python-rosdep       python3-colcon-common-extensions     && rm -rf /var/lib/apt/lists/*
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 4b3a4f918702
INFO:Docker Client:Step 6/10 : RUN rosdep init
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> d760a8a8591e
INFO:Docker Client:Step 7/10 : COPY gather_rosdeps.sh /root/
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 77fee945736d
INFO:Docker Client:Step 8/10 : RUN mkdir -p /ws
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> d7c12a2077c3
INFO:Docker Client:Step 9/10 : WORKDIR /ws
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> e5eba9dc0d20
INFO:Docker Client:Step 10/10 : ENTRYPOINT ["/root/gather_rosdeps.sh"]
INFO:Docker Client: ---> Using cache
INFO:Docker Client: ---> 217a5be9d9fe
INFO:Docker Client:Successfully built 217a5be9d9fe
INFO:Docker Client:Successfully tagged ros_cross_compile:rosdep
INFO:Rosdep Gatherer:Running rosdep collector image on workspace /builds/ros_ws
INFO:Docker Client:+ '[' '!' -d ./src ']'
INFO:Docker Client:++ pwd
INFO:Docker Client:+ echo 'No src/ directory found at /ws, did you remember to mount your workspace?'
INFO:Docker Client:+ exit 1
INFO:Docker Client:No src/ directory found at /ws, did you remember to mount your workspace?
Traceback (most recent call last):
  File "/usr/local/bin/ros_cross_compile", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/ros_cross_compile.py", line 227, in main
    cross_compile_pipeline(args, data_collector, platform)
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/ros_cross_compile.py", line 214, in cross_compile_pipeline
    stage(platform, docker_client, ros_workspace_dir, options, data_collector)
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/dependencies.py", line 136, in __call__
    custom_data_dir=options.custom_data_dir)
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/dependencies.py", line 87, in gather_rosdeps
    volumes=volumes,
  File "/usr/local/lib/python3.6/dist-packages/ros_cross_compile/docker_client.py", line 145, in run_container
    image_name, exit_code, '', image_name, 'See above ^')
docker.errors.ContainerError: Command '' in image 'ros_cross_compile:rosdep' returned non-zero exit status 1: See above ^
Cleaning up file based variables 00:01
ERROR: Job failed: exit code 1

Notice that in the output of the tree command there is a src/ directory on the workspace.

System (please complete the following information)

For ROS 2 bugs - please attach a VCS.repos or Dockerfile if possible to help us reproduce the environment

Additional context

This works fine when run from a Bionic VM in VMWare.

emersonknapp commented 3 years ago

Am I reading correctly that you're running this from inside a Docker container? The tool itself uses Docker - so, if so, I think there's probably something going wrong with the Docker-in-Docker situation - maybe that mounting the workspace as a volume isn't actually happening...?

kjeremy commented 3 years ago

Yes that's correct that it should be using docker in docker. I'm not terribly familiar with docker in docker so am unsure how to debug.

On Thu, May 20, 2021, 4:05 PM Emerson Knapp @.***> wrote:

Am I reading correctly that you're running this from inside a Docker container? The tool itself uses Docker - so, if so, I think there's probably something going wrong with the Docker-in-Docker situation.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ros-tooling/cross_compile/issues/313#issuecomment-845440353, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBACRHYHMIROL4GHEKAU2TTOVTPTANCNFSM45HTESPA .

emersonknapp commented 3 years ago

Are you able to run in the host environment instead? I'm inclined to say "running ros_cross_compile from inside a container is not supported"

kjeremy commented 3 years ago

Unfortunately I cannot run in the host environment for production. After reading some stack overflow (https://stackoverflow.com/questions/50971417/docker-inside-docker-volume-is-mounted-but-empty) I think a solution may be to use named volumes for the workspace.

emersonknapp commented 3 years ago

I would be happy to review changes along these lines. However, I think you would have to create the named volume ahead of time - still from the host environment, could you do that once you're already in a container?

kjeremy commented 3 years ago

I got it working successfully using docker-in-docker with the following:

image: teracy/ubuntu:18.04-dind-19.03.3

services:
  - docker:19.03.3-dind

variables:
  # Disable TLS or we get SSLv1 errors. We shouldn't need this since we mount the /certs volume.
  DOCKER_TLS_CERTDIR: ""
  DOCKER_HOST: tcp://docker:2375

While it would be nice to support with the "sharing the docker socket approach" this is a kind of workaround.

kjeremy commented 3 years ago

I'm going to close this since it works in dind.

emersonknapp commented 3 years ago

Others may appreciate your findings as a section in the README - if you're willing to contribute that. Maybe something like a "Troubleshooting" section, under which there is a subsection for "No src/ directory found at /ws, did you remember to mount your workspace?"

ZamesNg commented 3 years ago

hi, is there more instruction?

kjeremy commented 3 years ago

@ZhamaoDra What I did was set up a gitlab-runner running in docker in privileged mode and used the .gitlab-ci.yml file in the troubleshooting section (#314). I'm not sure what else you are looking for.