search

S2-group / robot-runner

Tool for Automatically Executing Experiments on Robotics Software
MIT License
29 stars 6 forks source link
empirical-software-engineering robotics ros

readme

Robot-Runner

Robot Runner (RR) is a tool to automatically execute measurement-based experiments on robotics software.

The following scientific paper gives an overview about the main components, plugins, and configurations of Robot Runner: ICSE 2021 tool demo.

A short video, giving a brief explanation of Robot Runner, can be seen here. ICSE 2021 Tool demo

How to cite Robot Runner

If Robot Runner is helping your research, consider to cite it as follows, thank you!

@inproceedings{ICSE_2021,
  title={{Robot Runner: A Tool for Automatically Executing Experiments on Robotics Software}},
  author={Stan Swanborn and Ivano Malavolta},
  booktitle = {Proceedings of the ACM/IEEE 43rd International Conference on Software Engineering},
  year={2021},
  url= {https://github.com/S2-group/robot-runner/tree/master/documentation/ICSE_2021.pdf},
  organization={ACM}
}

Overview

The steps to start using Robot Runner are detailed below.

As visualized below, Robot Runner consists of the following components:

Overview of Robot Runner

Robot Runner is developed to be entirely independent from any communicational means or robotic system. This means specifically; that Robot Runner can be used with any robotic system, or any simulation software, using any form of communication (e.g. ROS1, ROS2, TCP Sockets, etc.).

RR offers an automation of the infrastructure overhead for measurement-based, empirical, experiments as a consequence of its design, as produced by the following design drivers:

Setup guide

To be able to use RR, some requirements on the system in question need to be met:

The user's system is ready to run RR if the abovementioned requirements are met. However, for the communication with a robotic system such means will need to be installed. As mentioned before, the user is entirely free in this choice as RR is independent from this.

However, as the Robotic Operating System (ROS) is the de-facto standard for robotics software, it was used during development in combination with a ROBOTIS TurtleBot3 (Burger).

For the installation of ROS, any version can be used. But during development, ROS2 Foxy and ROS1 Melodic were explicitly used.

Quick start

Now that the system has all the necessary components installed to run Robot Runner, and has robotic enabling software installed (such as ROS), a quick start with Robot Runner can be performed.

Creating configuration file

Python3.8 robot-runner/ config-create [directory]

When running this command, where [directory] is an optional argument, a new config file with some example code will be generated. The default location for this would be robot-runner/experiments/, otherwise the given directory will be used.

Setting up an experiment

Now that a new configuration file is available, the user can define the experiment. An experiment configuration can be defined using the provided experiment parameters and events.

The parameters

    name:                       str             = "mini_test"
    required_ros_version:       int             = 2
    required_ros_distro:        str             = "foxy"
    operation_type:             OperationType   = OperationType.AUTO
    time_between_runs_in_ms:    int             = 1000
    results_output_path:        Path             = Path("~/Documents/experiments")

Supporting information:

The events

    def __init__(self):
        """Executes immediately after program start, on config load"""
        EventSubscriptionController.subscribe_to_multiple_events([ 
            (RobotRunnerEvents.BEFORE_EXPERIMENT,   self.before_experiment), 
            (RobotRunnerEvents.START_RUN,           self.start_run),
            (RobotRunnerEvents.START_MEASUREMENT,   self.start_measurement),
            (RobotRunnerEvents.LAUNCH_MISSION,      self.launch_mission),
            (RobotRunnerEvents.STOP_MEASUREMENT,    self.stop_measurement),
            (RobotRunnerEvents.STOP_RUN,            self.stop_run),
            (RobotRunnerEvents.CONTINUE,            self.continue_experiment)
            (RobotRunnerEvents.POPULATE_RUN_DATA,   self.populate_run_data),
            (RobotRunnerEvents.AFTER_EXPERIMENT,    self.after_experiment)
        ])

    def create_run_table(self) -> List[Dict]:
        """Create and return the run_table here. A run_table is a List (rows) of dictionaries (columns), 
        representing each run robot-runner must perform"""
        run_table = RunTableModel(
            factors = [
                FactorModel("example_factor", ['example_treatment1', 'example_treatment2'])
            ],
            exclude_variations = [
                {"example_treatment1"},
                {"example_treatment1", "example_treatment2"}
            ],
            data_columns=["data_column1", "data_column2"]
        )
        run_table.create_experiment_run_table()
        return run_table.get_experiment_run_table()

    def before_experiment(self) -> None:
        """Perform any activity required before starting the experiment here"""

    def start_run(self, context: RobotRunnerContext) -> None:
        """Perform any activity required for starting the run here. 
        Activities before and after starting the run should also be performed here."""

    def start_measurement(self, context: RobotRunnerContext) -> None:
        """Perform any activity required to start the measurements"""

    def launch_mission(self, context: RobotRunnerContext) -> None:
        """Perform any activity interacting with the robotic
        system in question (simulated or real-life) here."""

    def stop_measurement(self, context: RobotRunnerContext) -> None:
        """Perform any activity required to stop the measurements"""

    def stop_run(self, context: RobotRunnerContext) -> None:
        """Perform any activity required for stopping the run here.
        Activities before and after stopping the run should also be performed here."""

    def populate_run_data(self, context: RobotRunnerContext) -> tuple:
        """Return the run data as a row for the output manager represented as a tuple"""

    def continue_experiment(self, context: RobotRunnerContext) -> None:
        """On return of this callback, Robot Runner continues with the orchestration of the experiment"""

    def after_experiment(self) -> None:
        """Perform any activity required after stopping the experiment here"""

Performing the experiment

Once the experiment has been defined by the user, as business logic setup in the shown event callbacks above, the experiment can be performed by Robot Runner. To do this, the user runs the following command:

python3.8 robot-runner/ experiment_config.py

After which Robot Runner will:

Examples

Robot Runner offer a simple example for a ROS1 based robotic system. The experiment was performed on a ROBOTIS TurtleBot3 specifically. The example experiment is called 'mini-mission' and can be found in the robot-runner/experiments/mini-mission/ folder.

The mini-mission, its execution and its output is explained in the video referenced at the beginning of this README.

Supporting Features

Robot Runner offers extensive supporting infrastructure, such as:

Publications using Robot Runner