Using multimaster with 2 identical robots and a ground station

[mattgiamou]

Hi, I'd like to use multimaster for a setup with 3 machines, each running their own ROS master. One of the machines is a ground station, while the other 2 are identical vehicles with many nodes and topics. I would only like a small subset of topics from each of the vehicles to be shared with the base station alone. On each of the vehicles, I launch:

<launch>
  <arg name="robot_name" default="robot0" />
  <node name="master_discovery" pkg="master_discovery_fkie" type="master_discovery" >
  </node>
  <!-- Launch the relay node -->
  <node pkg="topic_tools" type="relay" name="robot_relay" args="/shared_topic /$(arg robot_name)/shared_topic" />
</launch>

with a different "robot_name" argument for each since I want the base station to differentiate between this shared_topic. On the base station, I launch:

<launch>
  <node name="master_discovery" pkg="master_discovery_fkie" type="master_discovery" output="screen">
  </node>
  <node name="master_sync" pkg="master_sync_fkie" type="master_sync" output="screen">
    <rosparam param="sync_topics"> ['/robot0/shared_topic', '/robot1/shared_topic'] </rosparam>
  </node>
</launch>

which allows the base station to successfully receive /robot0/shared_topic OR /robot1/shared_topic when only one of the vehicle's has launched the first launch file. When I try to run both, I get the following warning before the nodes on the vehicle shutdown:

Reason given for shutdown: [new node registered with same name].

Each vehicle and the base station have their own ROS Master with their own local IP address as the ROS_MASTER_URI. I was under the impression that master_discovery simply makes the machines aware of one another, and only master_sync will cause collisions like this. Is there some argument that master_discovery needs like "ignore_hosts" or "ignore_nodes"? I only want the ground station to be aware of a few topics from each vehicle, but the fact that the vehicles have a large number of nodes with the same name prevents this. I realize I could namespace each vehicle so that the topics and nodes all have different names, but I was wondering if there was another solution with just arguments to the launch files I have here, or if I'm misunderstanding the default behaviour of master_discovery. Thanks!

[atiderko]

Hi, your understanding of the master_discovery and master_sync is right. The only thing is: for each topic a unique node must be registered in ROS-Master. In your case: the name of robot_relay must be different for each vehicle, e.g.:

<node pkg="topic_tools" type="relay" name="$(arg robot_name)_relay" args="/shared_topic /$(arg robot_name)/shared_topic" />

Hope, this will help! regards.

[mattgiamou]

Hi, thanks for the prompt response. I fixed the issue from earlier, and both vehicles' topics appear on the base station (via rostopic list and rqt_graph), but one of these topics is not receiving the messages. I have checked that both vehicles are in fact publishing the messages locally, but only one vehicle's messages are received at the base station.

Here is the output of master_sync_fkie running on the ground station:

roslaunch perception_launch multi_master.launch
... logging to /home/matt/.ros/log/3449c898-714a-11e6-b766-4485006624de/roslaunch-mattquad-base0-4587.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://192.168.0.103:39676/

SUMMARY
========

PARAMETERS
* /master_sync/sync_topics: ['/robot1/flight/o...
* /rosdistro: indigo
* /rosversion: 1.11.20

NODES
/
master_discovery (master_discovery_fkie/master_discovery)
master_sync (master_sync_fkie/master_sync)

ROS_MASTER_URI=http://localhost:11311

core service [/rosout] found
process[master_discovery-1]: started with pid [4609]
process[master_sync-2]: started with pid [4610]
[INFO] [WallTime: 1472849293.984138] ignore_hosts: []
[INFO] [WallTime: 1472849293.985837] sync_hosts: []
[INFO] [WallTime: 1472849293.986744] ROS Master URI: http://localhost:11311
[INFO] [WallTime: 1472849293.987342] sync_topics_on_demand: False
[WARN] [WallTime: 1472849293.991444] Master_discovery node appear not to running. Wait for topic with type 'MasterState.
[INFO] [WallTime: 1472849293.997858] Check the ROS Master[Hz]: 1
[INFO] [WallTime: 1472849293.998051] Heart beat [Hz]: 0.02
[INFO] [WallTime: 1472849293.998167] Active request after [sec]: 60
[INFO] [WallTime: 1472849293.998279] Remove after [sec]: 300
[INFO] [WallTime: 1472849293.998379] Robot hosts: []
[INFO] [WallTime: 1472849293.998485] Approx. mininum avg. network load: 1.36 bytes/s
[INFO] [WallTime: 1472849294.005656] Start RPC-XML Server at ('0.0.0.0', 11611)
[INFO] [WallTime: 1472849294.006169] Subscribe to parameter `/roslaunch/uris`
[INFO] [WallTime: 1472849294.010896] + Bind to unicast socket @(192.168.0.103:11511)
[INFO] [WallTime: 1472849294.011132] Listen for multicast at ('226.0.0.0', 11511)
[INFO] [WallTime: 1472849294.158089] Detected master discovery: http://192.168.0.103:11611
[INFO] [WallTime: 1472849294.173221] Added master with ROS_MASTER_URI=http://192.168.0.103:11311/
[INFO] [WallTime: 1472849294.212483] Detected master discovery: http://192.168.0.100:11611
[INFO] [WallTime: 1472849294.269407] Added master with ROS_MASTER_URI=http://192.168.0.100:11311/
[INFO] [WallTime: 1472849295.010849] listen for updates on /master_discovery/changes
[INFO] [WallTime: 1472849295.028562] Update ROS master list...
[INFO] [WallTime: 1472849295.043835] service 'list_masters' found on /master_discovery/list_masters
[INFO] [WallTime: 1472849295.054478] ignore_nodes: ['/*node_manager', '/*master_sync*', '/rosout', '/*zeroconf', '/*master_discovery*']
[INFO] [WallTime: 1472849295.057665] sync_nodes: []
[INFO] [WallTime: 1472849295.059911] ignore_topics: ['/rosout', '/rosout_agg']
[INFO] [WallTime: 1472849295.062493] sync_topics: ['/robot2/topic', '/robot1/topic']
[INFO] [WallTime: 1472849295.065630] ignore_services: ['/*get_loggers', '/*set_logger_level']
[INFO] [WallTime: 1472849295.067631] sync_services: []
[INFO] [WallTime: 1472849295.069257] ignore_type: ['bond/Status']
[INFO] [WallTime: 1472849295.070994] ignore_publishers: []
[INFO] [WallTime: 1472849295.073500] ignore_subscribers: []
[INFO] [WallTime: 1472849295.076239] do_not_sync: []
[INFO] [WallTime: 1472849325.077775] Update ROS master list...
[INFO] [WallTime: 1472849325.104527] service 'list_masters' found on /master_discovery/list_masters
[INFO] [WallTime: 1472849326.280228] Detected master discovery: http://192.168.0.104:11611
[INFO] [WallTime: 1472849326.381374] Added master with ROS_MASTER_URI=http://192.168.0.104:11311/
[INFO] [WallTime: 1472849326.386678] ignore_nodes: ['/*node_manager', '/*master_sync*', '/rosout', '/*zeroconf', '/*master_discovery*']
[INFO] [WallTime: 1472849326.389398] sync_nodes: []
[INFO] [WallTime: 1472849326.392334] ignore_topics: ['/rosout', '/rosout_agg']
[INFO] [WallTime: 1472849326.395483] sync_topics: ['/robot2/topic', '/robot1/topic']
[INFO] [WallTime: 1472849326.398778] ignore_services: ['/*get_loggers', '/*set_logger_level']
[INFO] [WallTime: 1472849326.401740] sync_services: []
[INFO] [WallTime: 1472849326.404431] ignore_type: ['bond/Status']
[INFO] [WallTime: 1472849326.407294] ignore_publishers: []
[INFO] [WallTime: 1472849326.410574] ignore_subscribers: []
[INFO] [WallTime: 1472849326.413792] do_not_sync: []
[INFO] [WallTime: 1472849328.214285] SyncThread[192.168.0.104] Horrible hack: create and delete publisher to trigger an update for subscribed topics: set(['/robot1/topic'])

Note that the base station has IP address 192.168.103, the vehicle which is able to send topics is 192.168.100 (it is launched first and its name is robot2), and the vehicle which can't send topics is 192.168.104 (it is launched second and is called robot1). When the second vehicle is launched, the "Horrible hack" message is displayed. Is this a symptom of the problem? Is there any documentation of a multi-vehicle system similar to mine? Should I be using nodes other than master_discovery and master_sync?

Thank you for all your help, I will keep examining the documentation to try and fix my problem, but any insight is appreciated!

[atiderko]

You're welcome!

I see, you set the ROS_MASTER_URI to localhost. You have to change this to use ROS with remote nodes. see ROS_MASTER_URI

[mattgiamou]

Ahhh thank you so much, that did the trick! I had read elsewhere that for single ROS master setups I should avoid setting ROS_MASTER_URI explicitly, but that made it become localhost. It works great now!

[sunil1349]

how can give my shared topics to sync node from launch file