RQ1 Satisfaction

[schmerl]

Let's use this issue to keep track of the activities we need to do for RQ1. As I understand it, we need to:

• [ ] Build images for various robots - with interesting architectures (not just putting the robot in some world, but doing something)

  • [ ] Autorally
  • [x] Turtlebot
  • [ ] Turtlebot3
  • [ ] Autoware
  • [x] Husky
  • [ ] Fetch
  • [ ] (Add or remove others)

• [ ] Run recovery to understand the missing models. For each robot, let's keep track in this issue of (a) the number of models we had to handwrite because of not being in C++ (b) any issues that we had to fix to get the static recovery working (c) the number of C++ components we had to hand craft because we couldn't recover them and (i) whether it is a ROS core component or (ii) a component that is provided by that robot

• [ ] Run recovery for data collection of RQ1 in two phases

  • [ ] Component level accuracy
  • [ ] Architecture level accuracy

• [x] Build infrastructure for assessing component-level accuracy [@ChrisTimperley]

  • [x] Automatically obtain a list of all C++ nodes and nodelets (for which source is available) in a given image [@ChrisTimperley]
  • [x] Save config files with a complete set of node sources to the experiment directory (e.g., config-with-sources.yml). This will help us with checking that the node sources are complete and correct [@ChrisTimperley]
  • [x] Run rosdiscover recover on each node/nodelet [@ChrisTimperley]

• [ ] Build infrastructure for assessing architecture-level accuracy

[schmerl]

Besides this, are we planning to do a depth first approach or breadth first, i.e., get all the containers up and running then move onto the next step, or get autorally up and running and then do recovering and model creation and then assessment on that, then repeat with another container, and so on?

[ChrisTimperley]

I think that a depth-first approach is probably the best use of our time. Plus we're close to being finished with the pipeline. The biggest TODO item is to figure out the best way that we should obtain the ground truth architecture for a given system. Perhaps we can create a separate issue for that task specifically and discuss a few different approaches before settling on one?

[schmerl]

For architecture-level accuracy, I guess there are two options on the table:

1. Manually run the robot in the set of containers, and use rosdiscover observe to generate the architecture in Acme. We'd then need to understand the difference between the observed and recovered architecture. Let me see if I can find the work we did on architecture differenciing over a decade ago.
2. Manually check the architecture connections that are generated by rosdiscover against the components that are output.

1 is probably the best, but I'm not sure how the differencing will actually go.

[ChrisTimperley]

For generating the ground truth, it would be good for a bunch of reasons (e.g., reproducibility, sanity checking, scalability) if we could run the experiments either automatically or semi-automatically. We almost certainly want to use the same launch config to bring up the system, otherwise we're not comparing apples to apples.

As @schmerl pointed out, though, we may want to do more than simply launching the robot. E.g., we may want to instruct it to follow some basic instructions, navigate to a waypoint, etc. At a minimum, we should provide a description of this task for each RQ1 system in the corresponding experiment.yml file. Potentially, we could automate this process by allowing each RQ1 experiment.yml to specify an inline bash script for interacting with the robot. We would likely need to use sleep/timeouts rather than waiting for events to occur (e.g., reaching a particular destination), but that shouldn't be too much trouble.

run-script: >
  rosservice call ...
  sleep 30s

[ChrisTimperley]

Regardless of approach, we'll also want to update the observe command to:

• repeatedly check the run-time architecture at a fixed interval; add any missing connectors to the "output" architecture
• block until the user instructs the recording to stop (we'll have to tweak this slightly for automated approaches)

[schmerl]

The periodic observation has been implemented. Currently, the architecture isn't emitted until the observation period has finished (if --num-observations is specified, then the number of observations; if it isn't specified, then until Ctrl-C is pressed.

[ChrisTimperley]

Related to the second item in this list, I've created a separate issue: https://github.com/rosqual/rosdiscover-evaluation/issues/31