Improve distributed clock

[maurerle]

• adds method wait_all_online which waits until all the given agents in a DistributedClockManager are online. This removes the need to start the agent before the manager and makes it possible to run in docker/k8s tasks.

• this also revealed an interesting timing problem in mqtt when run with

timing problem

pytest tests/integration_tests/test_single_container_termination.py --durations=0

1.12s call     tests/integration_tests/test_single_container_termination.py::test_distribute_ping_pong_ts_mqtt
0.96s call     tests/integration_tests/test_single_container_termination.py::test_distribute_time_mqtt
0.81s call     tests/integration_tests/test_single_container_termination.py::test_send_current_time_mqtt
0.71s call     tests/integration_tests/test_single_container_termination.py::test_distribute_ping_pong_mqtt
0.14s call     tests/integration_tests/test_single_container_termination.py::test_termination_single_container
0.05s call     tests/integration_tests/test_single_container_termination.py::test_distribute_ping_pong_tcp
0.02s call     tests/integration_tests/test_single_container_termination.py::test_send_current_time_tcp
0.01s call     tests/integration_tests/test_single_container_termination.py::test_distribute_ping_pong_ts_tcp
0.01s call     tests/integration_tests/test_single_container_termination.py::test_distribute_time_tcp

First of all, one can see that the use of MQTT is slower than TCP by an order of magnitudes in these roundtrip tests.

The test test_distribute_ping_pong_ts uses the distributed clock to synchronize the clock of distributed containers. When using MQTT, this has a small delay of 40ms between every time update, while I did not find out where this comes from, even when looking at the code of paho-mqtt and spending a few hours on that. The test test_distribute_ping_pong_mqtt does not use a distributed clock, but directly ping pong using handle message and subscribe.

The delay probably comes from the synchronization of asyncio and the paho-mqtt event loop.

In general, the TCP connection method performs much better, when sending hundreds of roundtrips between two agents in different containers.

[rcschrg]

Thank you for this improvement (and many other improvements)! I am quite interested. Do you use MQTT for your project? Lately, we have felt a little bit out of touch with the MQTT part of mango, as we mostly use TCP or simulated communication.

[maurerle]

I want to support a way of communicating between agents, without having to open a port on each agent, that's where MQTT comes into play. Other than that, performance studies show that it is rather inferior to the TCP websocket connection. But I am still using it and compare it in some distributed tests, so there might be some more to it in the future :)