CI: Add proper testing for distributed dataflows

dora-rs / dora

DORA (Dataflow-Oriented Robotic Application) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.

https://dora-rs.ai

Apache License 2.0

1.35k stars 68 forks source link

CI: Add proper testing for distributed dataflows #475

Open phil-opp opened 2 months ago

phil-opp commented 2 months ago

Our current CI test runs the coordinator and two daemons on the same machine. To ensure that dora is able to communicate across machines, we would need to use three different machines with different IPs. I'm not sure if something like this is possible on GitHub actions, so this issue is mostly a reminder to look into this further.

Notes:

@haixuanTao mentioned the https://github.com/haythem/public-ip in https://github.com/dora-rs/dora/pull/471#discussion_r1568389104

Michael-J-Ward commented 2 months ago

As far as I can tell, the examples runners in CI serve as dora's integration tests. Run Completes = Green, but nothing verifies the output of the run - is that correct?

Are any of them deterministic enough that we could throw a snapshot tests on the output / logs - or otherwise test "successful end state"?

phil-opp commented 2 months ago

Yes, we don't very the output, as it's not deterministic enough. We just check the exit code of the process, which should be != 0 when an error occurs. We also have some examples with assertions, for example for verifying that the ROS2 turtle is at a different position at the end of the test (to ensure that it moved).