[Other issue]: dockerised fleet_adapter unable to find dockerised RMF Core

[cardboardcode]

Before proceeding, is there an existing issue or discussion for this?

• [X] I have done a search for similar issues and discussions.

Description

Issue Description :spiral_notepad:

In an attempt to dockerise fleet_adapter_template as well as RMF Core in order to have these 2 RMF components talk to each other from different environment, the following error is encountered:

Error Abstract :eye:

what():  Invalid rmf_traffic_msgs/ScheduleQuerySpacetime type [0]

Raw

Click here for more details

[rmf_traffic_schedule-1] terminate called after throwing an instance of 'std::runtime_error' [rmf_traffic_schedule-1] what(): Invalid rmf_traffic_msgs/ScheduleQuerySpacetime type [0] [rmf_traffic_schedule-1] Stack trace (most recent call last): [rmf_traffic_schedule-1] #23 Object "", at 0xffffffffffffffff, in [rmf_traffic_schedule-1] #22 Object "/opt/rmf/install/rmf_traffic_ros2/lib/rmf_traffic_ros2/rmf_traffic_schedule", at 0x402214, in _start [rmf_traffic_schedule-1] #21 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7ef3e21b1e3f, in __libc_start_main [rmf_traffic_schedule-1] #20 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7ef3e21b1d8f, in [rmf_traffic_schedule-1] #19 Object "/opt/rmf/install/rmf_traffic_ros2/lib/rmf_traffic_ros2/rmf_traffic_schedule", at 0x402536, in main [rmf_traffic_schedule-1] #18 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7ef3e26edc8e, in rclcpp::spin(std::shared_ptr) [rmf_traffic_schedule-1] #17 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7ef3e26edb94, in rclcpp::spin(std::shared_ptr) [rmf_traffic_schedule-1] #16 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7ef3e26ed97f, in rclcpp::executors::SingleThreadedExecutor::spin() [rmf_traffic_schedule-1] #15 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7ef3e26e60c5, in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) [rmf_traffic_schedule-1] #14 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7ef3e26e5d59, in rclcpp::Executor::execute_service(std::shared_ptr) [rmf_traffic_schedule-1] #13 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7ef3e26e8375, in [rmf_traffic_schedule-1] #12 Object "/opt/rmf/install/rmf_traffic_ros2/lib/librmf_traffic_ros2.so", at 0x7ef3e310e74b, in rclcpp::Service::handle_request(std::shared_ptr, std::shared_ptr) [rmf_traffic_schedule-1] #11 Object "/opt/rmf/install/rmf_traffic_ros2/lib/librmf_traffic_ros2.so", at 0x7ef3e311082f, in rclcpp::AnyServiceCallback::dispatch(std::shared_ptr > const&, std::shared_ptr const&, std::shared_ptr > >) [rmf_traffic_schedule-1] #10 Object "/opt/rmf/install/rmf_traffic_ros2/lib/librmf_traffic_ros2.so", at 0x7ef3e30fe23d, in std::_Function_handler, std::shared_ptr > >, std::shared_ptr > >), rmf_traffic_ros2::schedule::ScheduleNode::setup_query_services()::$_2>::_M_invoke(std::_Any_data const&, std::shared_ptr&&, std::shared_ptr > >&&, std::shared_ptr > >&&) [rmf_traffic_schedule-1] #9 Object "/opt/rmf/install/rmf_traffic_ros2/lib/librmf_traffic_ros2.so", at 0x7ef3e30f18e1, in rmf_traffic_ros2::schedule::ScheduleNode::register_query(std::shared_ptr const&, std::shared_ptr > > const&, std::shared_ptr > > const&) [rmf_traffic_schedule-1] #8 Object "/opt/rmf/install/rmf_traffic_ros2/lib/librmf_traffic_ros2.so", at 0x7ef3e31f6710, in rmf_traffic_ros2::convert(rmf_traffic_msgs::msg::ScheduleQuery_ > const&) [rmf_traffic_schedule-1] #7 Object "/opt/rmf/install/rmf_traffic_ros2/lib/librmf_traffic_ros2.so", at 0x7ef3e31f57d3, in rmf_traffic_ros2::convert(rmf_traffic_msgs::msg::ScheduleQuerySpacetime_ > const&) [rmf_traffic_schedule-1] #6 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7ef3e24814d7, in __cxa_throw [rmf_traffic_schedule-1] #5 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7ef3e2481276, in std::terminate() [rmf_traffic_schedule-1] #4 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7ef3e248120b, in [rmf_traffic_schedule-1] #3 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7ef3e2475b9d, in [rmf_traffic_schedule-1] #2 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7ef3e21b07f2, in abort [rmf_traffic_schedule-1] #1 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7ef3e21ca475, in raise [rmf_traffic_schedule-1] #0 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7ef3e221e9fc, in pthread_kill [rmf_traffic_schedule-1] Aborted (Signal sent by tkill() 78 0) [ERROR] [rmf_traffic_schedule-1]: process has died [pid 78, exit code -6, cmd '/opt/rmf/install/rmf_traffic_ros2/lib/rmf_traffic_ros2/rmf_traffic_schedule --ros-args -r __node:=rmf_traffic_schedule_primary --params-file /tmp/launch_params_9tc49n51']. [schedule_visualizer_node-4] [INFO] [1718244053.828736221] [schedule_data_node]: Requesting new schedule update because update timed out [schedule_visualizer_node-4] [INFO] [1718244053.828795097] [schedule_data_node]: [rmf_traffic_ros2::MirrorManager::request_update] Requesting changes for query ID [1] since beginning of recorded history [schedule_visualizer_node-4] [WARN] [1718244053.828830466] [schedule_data_node]: [MirrorManager::request_update] Waiting for change request service to reconnect for [727748c5-e9c3-4fa5-98e0-bd10cfd28bfd] before sending a change request.

Steps To Reproduce :books:

Follow the steps to recreate the error encountered:

Environment :bookmark_tabs:

• Ubuntu 22.04.4 LTS
• ROS 2 Humble
• Docker version 26.1.3, build b72abbb

1. Create the following directory with the files below:

cd $HOME

mkdir -p test_docker_rmf_core_fleet_adapter/map && cd test_docker_rmf_core_fleet_adapter/map

2. Create or Download the following files into the directory /map:

test.building.yaml :file_folder:

coordinate_system: reference_image
crowd_sim:
  agent_groups:
    - {agents_name: [], agents_number: 0, group_id: 0, profile_selector: external_agent, state_selector: external_static, x: 0, y: 0}
  agent_profiles:
    - {ORCA_tau: 1, ORCA_tauObst: 0.40000000000000002, class: 1, max_accel: 0, max_angle_vel: 0, max_neighbors: 10, max_speed: 0, name: external_agent, neighbor_dist: 5, obstacle_set: 1, pref_speed: 0, r: 0.25}
  enable: 0
  goal_sets: []
  model_types: []
  obstacle_set: {class: 1, file_name: L1_navmesh.nav, type: nav_mesh}
  states:
    - {final: 1, goal_set: -1, name: external_static, navmesh_file_name: ""}
  transitions: []
  update_time_step: 0.10000000000000001
graphs:
  {}
levels:
  L1:
    doors:
      - [17, 18, {motion_axis: [1, start], motion_degrees: [3, 90], motion_direction: [2, 1], name: [1, bathroom_door], plugin: [1, normal], right_left_ratio: [3, 1], type: [1, hinged]}]
      - [11, 10, {motion_axis: [1, start], motion_degrees: [3, 90], motion_direction: [2, -1], name: [1, bedroom_door], plugin: [1, normal], right_left_ratio: [3, 1], type: [1, hinged]}]
      - [12, 19, {motion_axis: [1, start], motion_degrees: [3, 90], motion_direction: [2, 1], name: [1, bedroom_doord], plugin: [1, normal], right_left_ratio: [3, 1], type: [1, hinged]}]
      - [2, 7, {motion_axis: [1, start], motion_degrees: [3, 90], motion_direction: [2, -1], name: [1, hall_door], plugin: [1, normal], right_left_ratio: [3, 1], type: [1, hinged]}]
    drawing:
      filename: default_floor_layout.png
    elevation: 0
    floors:
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [3, 2, 18, 17, 16, 15]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [8, 14, 13, 11, 10, 9]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [14, 6, 20, 19, 12, 13]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [20, 19, 12, 4, 5]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [16, 4, 15]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [2, 7, 8, 9, 16, 17, 18]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [16, 9, 4]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [9, 10, 4]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [10, 11, 4]
      - parameters: {ceiling_scale: [3, 1], ceiling_texture: [1, blue_linoleum], indoor: [2, 0], texture_name: [1, blue_linoleum], texture_rotation: [3, 0], texture_scale: [3, 1]}
        vertices: [11, 12, 4]
    lanes:
      - [23, 24, {bidirectional: [4, true], demo_mock_floor_name: [1, ""], demo_mock_lift_name: [1, ""], graph_idx: [2, 0], orientation: [1, ""], speed_limit: [3, 0]}]
      - [24, 25, {bidirectional: [4, true], demo_mock_floor_name: [1, ""], demo_mock_lift_name: [1, ""], graph_idx: [2, 0], orientation: [1, ""], speed_limit: [3, 0]}]
      - [25, 26, {bidirectional: [4, true], demo_mock_floor_name: [1, ""], demo_mock_lift_name: [1, ""], graph_idx: [2, 0], orientation: [1, ""], speed_limit: [3, 0]}]
      - [26, 23, {bidirectional: [4, true], demo_mock_floor_name: [1, ""], demo_mock_lift_name: [1, ""], graph_idx: [2, 0], orientation: [1, ""], speed_limit: [3, 0]}]
    layers:
      test_apartment:
        color: [1, 0, 0, 0.5]
        filename: ""
        transform:
          scale: 0.01
          translation_x: 0
          translation_y: 0
          yaw: 0
        visible: false
    measurements:
      - [0, 1, {distance: [3, 4]}]
    models:
      - {dispensable: false, model_name: OpenRobotics/OfficeChairGrey, name: OpenRobotics/OfficeChairGrey, static: true, x: 358.392, y: 317.47300000000001, yaw: -3.1183000000000001, z: 0}
      - {dispensable: false, model_name: OpenRobotics/OfficeChairGrey, name: OpenRobotics/OfficeChairGrey, static: true, x: 402.839, y: 316.76799999999997, yaw: -3.1246, z: 0}
      - {dispensable: false, model_name: OpenRobotics/Table, name: OpenRobotics/Table, static: true, x: 382.21499999999997, y: 292.65699999999998, yaw: -3.1328, z: -0.29999999999999999}
    vertices:
      - [32.286000000000001, 47.747999999999998, 0, ""]
      - [204.05600000000001, 47.747999999999998, 0, ""]
      - [35.155999999999999, 166.27699999999999, 0, ""]
      - [33.683999999999997, 31.550999999999998, 0, ""]
      - [495.28500000000003, 31.550999999999998, 0, ""]
      - [495.28500000000003, 562.35599999999999, 0, ""]
      - [32.947000000000003, 562.35599999999999, 0, ""]
      - [32.947000000000003, 219.28399999999999, 0, ""]
      - [33.683999999999997, 225.17400000000001, 0, ""]
      - [228.77799999999999, 225.17400000000001, 0, ""]
      - [228.77799999999999, 306.15600000000001, 0, ""]
      - [229.51499999999999, 355.48200000000003, 0, ""]
      - [229.51499999999999, 367.26100000000002, 0, ""]
      - [227.30600000000001, 361.37200000000001, 0, ""]
      - [33.683999999999997, 361.37200000000001, 0, ""]
      - [204.803, 33.387, 0, ""]
      - [204.803, 166.32900000000001, 0, ""]
      - [198.18899999999999, 166.32900000000001, 0, ""]
      - [149.24600000000001, 166.27699999999999, 0, ""]
      - [229.71899999999999, 413.57299999999998, 0, ""]
      - [229.71899999999999, 562.26700000000005, 0, ""]
      - [475.67700000000002, 564.06399999999996, 0, ""]
      - [432.447, 564.06399999999996, 0, ""]
      - [292.07600000000002, 388.02300000000002, 0, ""]
      - [288.548, 507.95800000000003, 0, ""]
      - [448.69600000000003, 514.30700000000002, 0, ""]
      - [450.81200000000001, 383.79000000000002, 0, ""]
    walls:
      - [2, 3, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [3, 4, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [4, 5, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [5, 6, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [6, 7, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [8, 9, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [9, 10, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [11, 12, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [13, 14, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [15, 16, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [16, 17, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [2, 18, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
      - [19, 20, {alpha: [3, 1], texture_height: [3, 2.5], texture_name: [1, default], texture_scale: [3, 1], texture_width: [3, 1]}]
lifts: {}
name: test_workspace

3. Run dockerised RMF Core using the commands below:

cd $HOME/test_docker_rmf_core_fleet_adapter

docker run -it --rm \
--name rmf_core_c \
-v $(pwd)/map:/opt/rmf/map \
ghcr.io/open-rmf/rmf_deployment_template/rmf-simulation:latest \
bash -c "ros2 launch rmf_demos common.launch.xml \
config_file:=/opt/rmf/map/test.building.yaml \
headless:=1 \
use_rmf_panel:=0 \
server_uri:=ws://localhost:8000/_internal"

4. Prepare the fleet_adapter_template using the commands below:

   cd $HOME/test_docker_rmf_core_fleet_adapter

   git clone https://github.com/open-rmf/fleet_adapter_template --branch humble --single-branch --depth 1

5. Create file Dockerfile in the newly created directory /fleet_adapter_template:

cd $HOME/test_docker_rmf_core_fleet_adapter/fleet_adapter_template

Dockerfile :file_folder:

FROM ghcr.io/open-rmf/rmf/rmf_demos:latest

RUN apt-get update \
  && apt-get remove python3-starlette -y \
  && apt-get install -y \
    cmake \
    curl \
    git \
    python3-colcon-common-extensions \
    python3-vcstool \
    qtbase5-dev qtchooser qt5-qmake qtbase5-dev-tools \
    wget \
    python3-pip \
  && pip3 install flask-socketio fastapi uvicorn nudged \
  && rm -rf /var/lib/apt/lists/*

# setup keys
WORKDIR /fleet_adapter_ws
COPY . src
RUN rosdep update --rosdistro $ROS_DISTRO

# This replaces: wget https://raw.githubusercontent.com/open-rmf/rmf/main/rmf.repos
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
    && apt-get upgrade -y \
    && rosdep update \
    && rosdep install --from-paths src --ignore-src --rosdistro $ROS_DISTRO -yr \
    && rm -rf /var/lib/apt/lists/*

# colcon compilation
RUN . /opt/ros/$ROS_DISTRO/setup.sh \
  && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release

# cleanup
RUN sed -i '$isource "/fleet_adapter_ws/install/setup.bash"' /ros_entrypoint.sh

ENTRYPOINT ["/ros_entrypoint.sh"]
CMD ros2 run fleet_adapter_template fleet_adapter -c /fleet_adapter_ws/src/fleet_adapter_template/config.yaml -n /fleet_adapter_ws/src/fleet_adapter_template/0.yaml -s ws://localhost:8000

6. Create file 0.yaml navigation graph file in the following directory /fleet_adapter_template/:

cd $HOME/test_docker_rmf_core_fleet_adapter/fleet_adapter_template/

0.yaml :file_folder:

building_name: dummy
levels:
  L1:
    lanes:
    - - 0
      - 1
      - {door_name: lab_entrance_door_right, is_bidirectional: false, speed_limit: 0}
    - - 1
      - 2
      - {is_bidirectional: false, speed_limit: 0}
    - - 2
      - 3
      - {door_name: lab_entrance_door_left, is_bidirectional: false, speed_limit: 0}
    - - 3
      - 0
      - {is_bidirectional: false, speed_limit: 0}
    vertices:
    - - 1.7782732024514007
      - -6.948913915578813
      - {name: ''}
    - - 24.08566297204254
      - -6.872960068283951
      - {name: ''}
    - - 24.11095590340457
      - -7.480665899851638
      - {name: ''}
    - - 1.7782732024514007
      - -7.607243136474944
      - {name: ''}

7. Build dockerised fleet_adapter_template using commands below:

   cd $HOME/test_docker_rmf_core_fleet_adapter/fleet_adapter_template

   docker build -t fleet_adapter_template:latest .

8. Run dockerised fleet_adapter_template using commands below:

   docker run -it --rm  \
   --network host \
   --name fleet_adapter_template_c \
   fleet_adapter_template:latest

Expected Behaviour :green_circle:

fleet_adapter_template is able to find RMF Schedule Core but gives error being unable to connect to connect to non-existent robot API server at http://127.0.0.1:8080, as specified in config.yaml.

Actual Behaviour :red_circle:

Upon running dockerised fleet_adapter_template using the steps above, dockerised RMF Core crashes with the aforementioned error.

Remarks

Appreciate any constructive help/feedback on this issue. :blush: :pray:

The rationale for this form of setup is for better system scalability where an RMF Deployment would have a robot's fleet adapter running in a different environment and server from RMF Core which should be running in cloud with Dashboard and API Server. This is in contrast to the many online tutorial examples of RMF deployment which assumes fleet adapter is always running in the same environment as RMF Core.

[cardboardcode]

Update :speech_balloon:

Been continuing to debug on this issue myself for the past week. Arrived at a short-term conclusion with a Dirty Workaround to avoid the issue as well as narrowing of the Cause of Issue.

What I Have Done :hammer:

The Cause of Issue has been narrowed down by incrementally shaving away at instantiated ROS 2 packages in common.launch.xml under rmf_demos. The following is a minimal common.launch.xml file which gives the core error reported as stated above.

<?xml version='1.0' ?>

<launch>

  <arg name="use_sim_time" default="false" description="Use the /clock topic for time to sync with simulation"/>

  <!-- Traffic Schedule  -->
  <node pkg="rmf_traffic_ros2" exec="rmf_traffic_schedule" output="both" name="rmf_traffic_schedule_primary">
    <param name="use_sim_time" value="$(var use_sim_time)"/>
  </node>

</launch>

Comparing this to a working example of .launch.xml that launches RMF Core in a manner that does not crash upon the launch of fleet_adapter_template, there is no difference in the way rmf_traffic_schedule was called.

Deduction :mag:

Therefore, the only difference between Faulty Setup and Working Setup seems to :red_circle: due to the base docker image used :red_circle: .

Working Setup uses ghcr.io/open-rmf/rmf/rmf_demos:latest when it was still based in ROS 2 Humble. Faulty Setup uses ghcr.io/open-rmf/rmf_deployment_template/rmf-simulation:latest which is based in ROS 2 Humble.

The rmf_traffic_schedule used in ghcr.io/open-rmf/rmf_deployment_template/rmf-simulation:latest, for lack of better explaination, is just different and should not be used, at least as of this writing and author's personal understanding.

Dirty Workaround :adhesive_bandage:

To avoid the core error reported in this thread, would recommend to use ghcr.io/open-rmf/rmf/rmf_demos:latest as a base docker image to quickly instantiate an RMF Core for development purposes.

Closing since resolving this issue may not be useful in long run, given developers are probably not going to use ghcr.io/open-rmf/rmf_deployment_template/rmf-simulation:latest for their official deployment of an RMF Core.