Launch nodes on different machines that share one roscore

Hi,

first of all thanks a lot for sharing and maintaining this awesome tool!

I have a question about how to use node_manager with the current setup we have (and I'm not sure if this is possible).

We have 4 PCs (see pictures):

A runs its own roscore and node_manager. It's connected over WiFi to the other PCs.
B runs its own roscore and it's connected to the router.
C has ROS_MASTER_URI=http://B:11311 set in its .bashrc and is connected to the router.
D has ROS_MASTER_URI=http://B:11311 set in its .bashrc and is connected to the router.

Given this setup (that cannot be easily changed) I'm able to sync A and B using node_manager, but I don't know how to include C and D, while keeping their ROS_MASTER_URI "pointing" to B. The goal is to launch specific launch files on B, C, and D from A. Is this possible with node_manager, without having to run roscores on C and D?

Thanks for your help. Marco.

Multimaster setup

Hi Marco,

this is not the main focus of the node manager, however I added support with last commit. In principle it is now possible but is not full supported in visualization.

Here are steps how to start:

start manually on hosts C and D node_manager_daemon with different names e.g.: rosrun fkie_node_manager_daemon node_manager_daemon __ns:=c __name:=node_manager_daemon
- you can also use node manager to run this nodes:

modify your launch file:
- add machine tag, e.g.: <machine name="C" address="C"/>
- add machine to all nodes which should be started on hosts C or D, e.g.: <node machine="C" ... />

Now, if you start the nodes, they will be launched on specified host. /!\ Currently the nodes with machine tag are not located on specified host. I will try to fix it.

Hi @atiderko ,

thanks a lot for your fast reply and even faster commit, amazing! I'd like to test the latest feature and therefore am installing node_manager from source, following the instructions here.

I am not sure if the error I get is related to your latest commits, or if I am doing something wrong:

user@host:~/catkin_ws$ node_manager

Traceback (most recent call last):
  File "/home/marcot/catkin_ws/devel/bin/node_manager", line 15, in <module>
    exec(compile(fh.read(), python_script, 'exec'), context)
  File "/home/marcot/catkin_ws/src/multimaster_fkie/fkie_node_manager/nodes/node_manager", line 5, in <module>
    import fkie_node_manager
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_node_manager/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 45, in <module>
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_node_manager_daemon/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 47, in <module>
  File "/home/marcot/catkin_ws/src/multimaster_fkie/fkie_node_manager_daemon/src/fkie_node_manager_daemon/server.py", line 40, in <module>
    import fkie_multimaster_msgs.grpc.file_pb2_grpc as fgrpc
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_multimaster_msgs/grpc/file_pb2_grpc.py", line 4, in <module>
    import file_pb2 as file__pb2
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_multimaster_msgs/grpc/file_pb2.py", line 23, in <module>
    \x10\x03\"\xb8\x01\n\x0cReturnStatus\x12\x0c\n\x04\x63ode\x18\x01 \x01(\x11\x12\x12\n\nerror_code\x18\x02 \x01(\x11\x12\x11\n\terror_msg\x18\x03 \x01(\t\x12\x12\n\nerror_file\x18\x04 \x01(\t\"_\n\nStatusType\x12\x06\n\x02OK\x10\x00\x12\t\n\x05\x45RROR\x10\x01\x12\x0c\n\x08IO_ERROR\x10\x02\x12\x0c\n\x08OS_ERROR\x10\x03\x12\x10\n\x0c\x43HANGED_FILE\x10\x04\x12\x10\n\x0cREMOVED_FILE\x10\x05\".\n\x13ListPackagesRequest\x12\x17\n\x0f\x63lear_ros_cache\x18\x01 \x01(\x08\"\x84\x01\n\x11ListPackagesReply\x12\x38\n\x06status\x18\x01 \x01(\x0b\x32(.fkie_multimaster_msgs.file.ReturnStatus\x12\x35\n\x05items\x18\x02 \x03(\x0b\x32&.fkie_multimaster_msgs.file.PackageObj\"(\n\nPackageObj\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04path\x18\x02 \x01(\t2\xd8\x06\n\x0b\x46ileService\x12v\n\x0eGetFileContent\x12\x31.fkie_multimaster_msgs.file.GetFileContentRequest\x1a/.fkie_multimaster_msgs.file.GetFileContentReply0\x01\x12{\n\x0fSaveFileContent\x12\x32.fkie_multimaster_msgs.file.SaveFileContentRequest\x1a\x30.fkie_multimaster_msgs.file.SaveFileContentReply(\x01\x30\x01\x12\x61\n\nCopyFileTo\x12).fkie_multimaster_msgs.file.CopyToRequest\x1a(.fkie_multimaster_msgs.file.ReturnStatus\x12]\n\x06Rename\x12).fkie_multimaster_msgs.file.RenameRequest\x1a(.fkie_multimaster_msgs.file.ReturnStatus\x12\x62\n\x08ListPath\x12+.fkie_multimaster_msgs.file.ListPathRequest\x1a).fkie_multimaster_msgs.file.ListPathReply\x12n\n\x0cListPackages\x12/.fkie_multimaster_msgs.file.ListPackagesRequest\x1a-.fkie_multimaster_msgs.file.ListPackagesReply\x12Z\n\x0c\x43hangedFiles\x12$.fkie_multimaster_msgs.file.PathList\x1a$.fkie_multimaster_msgs.file.PathList\x12\x62\n\x12GetPackageBinaries\x12&.fkie_multimaster_msgs.file.PackageObj\x1a$.fkie_multimaster_msgs.file.PathListb\x06proto3')
TypeError: __init__() got an unexpected keyword argument 'syntax'

In any case, thanks a lot for your valuable help! Marco.

it looks like a problem with version mismatch of protobuf and grpcio-tools.

Try to uninstall pip's protobuf and grpcio-tools and install with pip grpcio-tools version 0.14.

And if you installed python-protobuf with apt: uninstall or use in pip the same version...

@atiderko once again, thanks for your fast reply. Please let me knw if I should open a different issue for this.

I still get errors. What I tried was (no python-protobuf installed via apt):

sudo pip uninstall grpcio-tools
sudo pip uninstall protobuf
pip install grpcio-tools==0.14

I verified that:

$ pip show protobuf
Name: protobuf
Version: 3.8.0rc1

$ pip show grpcio-tools
Name: grpcio-tools
Version: 0.14.0

Then:

$ node_manager
Traceback (most recent call last):
  File "/home/marcot/catkin_ws/devel/bin/node_manager", line 15, in <module>
    exec(compile(fh.read(), python_script, 'exec'), context)
  File "/home/marcot/catkin_ws/src/multimaster_fkie/fkie_node_manager/nodes/node_manager", line 5, in <module>
    import fkie_node_manager
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_node_manager/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 45, in <module>
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_node_manager_daemon/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 47, in <module>
  File "/home/marcot/catkin_ws/src/multimaster_fkie/fkie_node_manager_daemon/src/fkie_node_manager_daemon/server.py", line 40, in <module>
    import fkie_multimaster_msgs.grpc.file_pb2_grpc as fgrpc
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_multimaster_msgs/grpc/file_pb2_grpc.py", line 4, in <module>
    import file_pb2 as file__pb2
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_multimaster_msgs/grpc/file_pb2.py", line 23, in <module>
    \x10\x03\"\xb8\x01\n\x0cReturnStatus\x12\x0c\n\x04\x63ode\x18\x01 \x01(\x11\x12\x12\n\nerror_code\x18\x02 \x01(\x11\x12\x11\n\terror_msg\x18\x03 \x01(\t\x12\x12\n\nerror_file\x18\x04 \x01(\t\"_\n\nStatusType\x12\x06\n\x02OK\x10\x00\x12\t\n\x05\x45RROR\x10\x01\x12\x0c\n\x08IO_ERROR\x10\x02\x12\x0c\n\x08OS_ERROR\x10\x03\x12\x10\n\x0c\x43HANGED_FILE\x10\x04\x12\x10\n\x0cREMOVED_FILE\x10\x05\".\n\x13ListPackagesRequest\x12\x17\n\x0f\x63lear_ros_cache\x18\x01 \x01(\x08\"\x84\x01\n\x11ListPackagesReply\x12\x38\n\x06status\x18\x01 \x01(\x0b\x32(.fkie_multimaster_msgs.file.ReturnStatus\x12\x35\n\x05items\x18\x02 \x03(\x0b\x32&.fkie_multimaster_msgs.file.PackageObj\"(\n\nPackageObj\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04path\x18\x02 \x01(\t2\xd8\x06\n\x0b\x46ileService\x12v\n\x0eGetFileContent\x12\x31.fkie_multimaster_msgs.file.GetFileContentRequest\x1a/.fkie_multimaster_msgs.file.GetFileContentReply0\x01\x12{\n\x0fSaveFileContent\x12\x32.fkie_multimaster_msgs.file.SaveFileContentRequest\x1a\x30.fkie_multimaster_msgs.file.SaveFileContentReply(\x01\x30\x01\x12\x61\n\nCopyFileTo\x12).fkie_multimaster_msgs.file.CopyToRequest\x1a(.fkie_multimaster_msgs.file.ReturnStatus\x12]\n\x06Rename\x12).fkie_multimaster_msgs.file.RenameRequest\x1a(.fkie_multimaster_msgs.file.ReturnStatus\x12\x62\n\x08ListPath\x12+.fkie_multimaster_msgs.file.ListPathRequest\x1a).fkie_multimaster_msgs.file.ListPathReply\x12n\n\x0cListPackages\x12/.fkie_multimaster_msgs.file.ListPackagesRequest\x1a-.fkie_multimaster_msgs.file.ListPackagesReply\x12Z\n\x0c\x43hangedFiles\x12$.fkie_multimaster_msgs.file.PathList\x1a$.fkie_multimaster_msgs.file.PathList\x12\x62\n\x12GetPackageBinaries\x12&.fkie_multimaster_msgs.file.PackageObj\x1a$.fkie_multimaster_msgs.file.PathListb\x06proto3')
TypeError: __init__() got an unexpected keyword argument 'syntax'

@marco-tranzatto my mistake, I was wrong...

on my Ubuntu 16.04 I have follow versions installed:

$ pip show protobuf
Name: protobuf
Version: 3.6.1

$ pip show grpcio-tools
Name: grpcio-tools
Version: 1.15.0

After update you should at least clean and rebuild fkie_multimaster_msgs package.

@atiderko thanks for checking. On Ubuntu 18.04, ROS Melodic I have now:

$ pip show grpcio-tools
Name: grpcio-tools
Version: 1.20.1
Summary: Protobuf code generator for gRPC
Home-page: https://grpc.io
Author: The gRPC Authors
Author-email: grpc-io@googlegroups.com
License: Apache License 2.0
Location: /home/marcot/.local/lib/python2.7/site-packages
Requires: protobuf, grpcio

$ pip show protobuf
Name: protobuf
Version: 3.7.1
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: None
Author-email: None
License: 3-Clause BSD License
Location: /home/marcot/.local/lib/python2.7/site-packages
Requires: six, setuptools

Then:

$ catkin clean fkie_multimaster_msgs
$ catkin clean fkie_node_manager
$ catkin build fkie_node_manager
$ source devel/setup.bash

But again:

$ node_manager
Traceback (most recent call last):
  File "/home/marcot/catkin_ws/devel/bin/node_manager", line 15, in <module>
    exec(compile(fh.read(), python_script, 'exec'), context)
  File "/home/marcot/catkin_ws/src/multimaster_fkie/fkie_node_manager/nodes/node_manager", line 5, in <module>
    import fkie_node_manager
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_node_manager/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 45, in <module>
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_node_manager_daemon/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 47, in <module>
  File "/home/marcot/catkin_ws/src/multimaster_fkie/fkie_node_manager_daemon/src/fkie_node_manager_daemon/server.py", line 40, in <module>
    import fkie_multimaster_msgs.grpc.file_pb2_grpc as fgrpc
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_multimaster_msgs/grpc/file_pb2_grpc.py", line 4, in <module>
    import file_pb2 as file__pb2
  File "/home/marcot/catkin_ws/devel/lib/python2.7/dist-packages/fkie_multimaster_msgs/grpc/file_pb2.py", line 23, in <module>
    \x10\x03\"\xb8\x01\n\x0cReturnStatus\x12\x0c\n\x04\x63ode\x18\x01 \x01(\x11\x12\x12\n\nerror_code\x18\x02 \x01(\x11\x12\x11\n\terror_msg\x18\x03 \x01(\t\x12\x12\n\nerror_file\x18\x04 \x01(\t\"_\n\nStatusType\x12\x06\n\x02OK\x10\x00\x12\t\n\x05\x45RROR\x10\x01\x12\x0c\n\x08IO_ERROR\x10\x02\x12\x0c\n\x08OS_ERROR\x10\x03\x12\x10\n\x0c\x43HANGED_FILE\x10\x04\x12\x10\n\x0cREMOVED_FILE\x10\x05\".\n\x13ListPackagesRequest\x12\x17\n\x0f\x63lear_ros_cache\x18\x01 \x01(\x08\"\x84\x01\n\x11ListPackagesReply\x12\x38\n\x06status\x18\x01 \x01(\x0b\x32(.fkie_multimaster_msgs.file.ReturnStatus\x12\x35\n\x05items\x18\x02 \x03(\x0b\x32&.fkie_multimaster_msgs.file.PackageObj\"(\n\nPackageObj\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04path\x18\x02 \x01(\t2\xd8\x06\n\x0b\x46ileService\x12v\n\x0eGetFileContent\x12\x31.fkie_multimaster_msgs.file.GetFileContentRequest\x1a/.fkie_multimaster_msgs.file.GetFileContentReply0\x01\x12{\n\x0fSaveFileContent\x12\x32.fkie_multimaster_msgs.file.SaveFileContentRequest\x1a\x30.fkie_multimaster_msgs.file.SaveFileContentReply(\x01\x30\x01\x12\x61\n\nCopyFileTo\x12).fkie_multimaster_msgs.file.CopyToRequest\x1a(.fkie_multimaster_msgs.file.ReturnStatus\x12]\n\x06Rename\x12).fkie_multimaster_msgs.file.RenameRequest\x1a(.fkie_multimaster_msgs.file.ReturnStatus\x12\x62\n\x08ListPath\x12+.fkie_multimaster_msgs.file.ListPathRequest\x1a).fkie_multimaster_msgs.file.ListPathReply\x12n\n\x0cListPackages\x12/.fkie_multimaster_msgs.file.ListPackagesRequest\x1a-.fkie_multimaster_msgs.file.ListPackagesReply\x12Z\n\x0c\x43hangedFiles\x12$.fkie_multimaster_msgs.file.PathList\x1a$.fkie_multimaster_msgs.file.PathList\x12\x62\n\x12GetPackageBinaries\x12&.fkie_multimaster_msgs.file.PackageObj\x1a$.fkie_multimaster_msgs.file.PathListb\x06proto3')
TypeError: __init__() got an unexpected keyword argument 'syntax'

@marco-tranzatto something is wrong with your install.

I intalled in Ubuntu 18.04 in VirtualBox, then:

pip install --user grpcio-tools
Collecting grpcio-tools
  Downloading https://files.pythonhosted.org/packages/ae/76/b681a298236bca5e54ae3672d505efe2620c816fa0a048ec8786619d0fa3/grpcio_tools-1.20.1-cp27-cp27mu-manylinux1_x86_64.whl (2.3MB)
    100% |████████████████████████████████| 2.3MB 315kB/s 
Collecting protobuf>=3.5.0.post1 (from grpcio-tools)
  Downloading https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 561kB/s 
Collecting grpcio>=1.20.1 (from grpcio-tools)
  Downloading https://files.pythonhosted.org/packages/71/f4/175dc28565a59860912c11d859d15d829b8f0e6fe0c8dc7405c6560d52e1/grpcio-1.20.1-cp27-cp27mu-manylinux1_x86_64.whl (2.1MB)
    100% |████████████████████████████████| 2.1MB 325kB/s 
Collecting six>=1.9 (from protobuf>=3.5.0.post1->grpcio-tools)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting setuptools (from protobuf>=3.5.0.post1->grpcio-tools)
  Downloading https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl (575kB)
    100% |████████████████████████████████| 583kB 1.1MB/s 
Collecting enum34>=1.0.4; python_version < "3.4" (from grpcio>=1.20.1->grpcio-tools)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting futures>=2.2.0; python_version < "3.2" (from grpcio>=1.20.1->grpcio-tools)
  Downloading https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Installing collected packages: six, setuptools, protobuf, enum34, futures, grpcio, grpcio-tools
Successfully installed enum34-1.1.6 futures-3.2.0 grpcio-1.20.1 grpcio-tools-1.20.1 protobuf-3.7.1 setuptools-41.0.1 six-1.12.0

... and it works. Try to remove protobuf, grpcio and `grpcio-tools and reinstall with

pip install --user grpcio-tools

and remove build and devel folder with catkin clean in your ros workspace.

Hi @atiderko

I found time to test your suggestions and so far it works as expected. Thanks a lot for being so fast to implement this extra feature!

For further reference, in case anybody else is interested in the same setup, these are some important points to keep in mind:

Use latest version of node manager, with the feature introduced as explained at the beginning of this thread.
When using the machine tag, remember to:
- Use RSA algorithm to copy the ssh key from one pc to another (applies for A to all the others, and from B to C and D). For example: ssh-copy-id -oHostKeyAlgorithms='ssh-rsa' user@host
- Use "env-loader script" (see example). On PCs C and D I put the following lines in the script:
  - source ~/catkin_ws/devel/setup.bash
  - export ROS_MASTER_URI=http://host-pc-a:11311
  - exec "$@"
Have one launch file to be launched on pc B that launches nodes on B, and uses one machine tag for C and one for D, to "distribute" nodes across machines.

Regards, Marco.

@marco-tranzatto thank you very much for your feedback!

fkie / multimaster_fkie

Launch nodes on different machines that share one roscore #104