use_internal_syncer in multi-node over socket protocol

Hi,

I'm trying to run a multi-node simulation with correct time sync. I'm hoping I can achieve this with socket protocol without having to set up hla.

I also need to have very regular sensor updates, so I was hoping to use use_internal_syncer/morse_sync to do this, which appears to be built for this purpose. Ideally, I'd like to have all nodes syncing to the 'real' clock as enforced by morse_sync.

However, it looks like use_internal_syncer (and morse_sync) does not behave correctly in multi-node mode.

Consider the following situation:

env.configure_multinode(    protocol='socket',
                            server_address='localhost',
                            server_port='65000',
                            distribution={
                                "node1": ['robot1'],
                                "node2": ['robot2'],
                            })
env.use_internal_syncer()

In this case, morse_sync will apply to node1 and node1 only, because use_internal_syncer forces you to use port 6000. Only the very first node will get a hold of port 6000, and others will not be able to use it.

As a hacky workaround, I tried something like:

sync_ports = {'node1': 6000, 'node2': 6001}
env.configure_multinode(    protocol='socket',
                            server_address='localhost',
                            server_port='65000',
                            distribution={
                                "node1": ['robot1'],
                                "node2": ['robot2'],
                            })
# exapnded version of env.use_internal_syncer()
env.configure_stream('socket', time_sync=True, sync_port=sync_ports[os.environ['MORSE_NODE']])

This does something closer to my expectation, in that independent morse_sync's apply to each node. However, the nodes do not sync to each other, which isn't surprising because the morse_sync's will be out of sync themselves.

In my mind, we could change morse_sync to support syncing multiple socket endpoints, which might allow syncing multiple nodes. Also, use_internal_syncer should take multi-node situation into account. What do you guys think?

*MORSE version: morse 1.4-183-gae45
Blender version: 2.79.0
Python version: 3.6.9

morse-simulator / morse

use_internal_syncer in multi-node over socket protocol #821