search

qcr / benchbot

BenchBot is a tool for seamlessly testing & evaluating semantic scene understanding tools in both realistic 3D simulation & on real robots
BSD 3-Clause "New" or "Revised" License
110 stars 12 forks source link
3d-simulation benchbot benchmarking evaluation nvidia-isaac real-robots robotics sim2real software-stack

readme

~ Our Robotic Vision Scene Understanding (RVSU) Challenge is live on EvalAI ~

~ BenchBot is now powered by NVIDIA Omniverse and Isaac Sim. We are aware of some issues, please report any you do encounter. ~

~ Our BenchBot tutorial is the best place to get started developing with BenchBot ~

BenchBot Software Stack

BenchBot project QUT Centre for Robotics Open Source Primary language License

benchbot_web

The BenchBot software stack is a collection of software packages that allow end users to control robots in real or simulated environments with a simple python API. It leverages the simple "observe, act, repeat" approach to robot problems prevalent in reinforcement learning communities (OpenAI Gym users will find the BenchBot API interface very similar).

BenchBot was created as a tool to assist in the research challenges faced by the semantic scene understanding community; challenges including understanding a scene in simulation, transferring algorithms to real world systems, and meaningfully evaluating algorithm performance. We've since realised, these challenges don't just exist for semantic scene understanding, they're prevalent in a wide range of robotic problems.

This led us to create version 2 of BenchBot with a focus on allowing users to define their own functionality for BenchBot through add-ons. Want to integrate your own environments? Plug-in new robot platforms? Define new tasks? Share examples with others? Add evaluation measures? This all now possible with add-ons, and you don't have to do anything more than add some YAML and Python files defining your new content!

The "bench" in "BenchBot" refers to benchmarking, with our goal to provide a system that greatly simplifies the benchmarking of novel algorithms in both realistic 3D simulation and on real robot platforms. If there is something else you would like to use BenchBot for (like integrating different simulators), please let us know. We're very interested in BenchBot being the glue between your novel robotics research and whatever your robot platform may be.

This repository contains the software stack needed to develop solutions for BenchBot tasks on your local machine. It installs and configures a significant amount of software for you, wraps software in stable Docker images (~50GB), and provides simple interaction with the stack through 4 basic scripts: benchbot_install, benchbot_run, benchbot_submit, and benchbot_eval.

System recommendations and requirements

The BenchBot software stack is designed to run seamlessly on a wide number of system configurations (currently limited to Ubuntu 18.04+). System hardware requirements are relatively high due to the software run for 3D simulation (e.g. NVIDIA Omniverse-powered Isaac Sim):

Having a system that meets the above hardware requirements is all that is required to begin installing the BenchBot software stack. The install script analyses your system configuration and offers to install any missing software components interactively. The list of 3rd party software components involved includes:

Managing your installation

Installation is simple:

u@pc:~$ git clone https://github.com/qcr/benchbot && cd benchbot
u@pc:~$ ./install

Any missing software components, or configuration issues with your system, should be detected by the install script and resolved interactively (you may be prompted to manually reboot and restart the install script). The installation asks if you want to add BenchBot helper scripts to your PATH. Choosing yes will make the following commands available from any directory: benchbot_install (same as ./install above), benchbot_run, benchbot_submit, benchbot_eval, and benchbot_batch.

The BenchBot software stack will frequently check for updates and can update itself automatically. To update simply run the install script again (add the --force-clean flag if you would like to install from scratch):

u@pc:~$ benchbot_install

If you decide to uninstall the BenchBot software stack, run:

u@pc:~$ benchbot_install --uninstall

There are a number of other options to customise your BenchBot installation, which are all described by running:

u@pc:~$ benchbot_install --help

Managing installed BenchBot add-ons

BenchBot installs a default set of add-ons, which is currently 'benchbot-addons/ssu' (and all of its dependencies declared here). But you can also choose to install a different set of add-ons instead. For example, the following will also install the 'benchbot-addons/data_collect' add-ons:

u@pc:~$ benchbot_install --addons benchbot-addons/ssu,benchbot-addons/data_collect

See the BenchBot Add-ons Manager's documentation for more information on using add-ons. All of our official add-ons can be found in our benchbot-addons GitHub organisation. We're open to adding add-ons contributed by our users to the official list as well.

Getting started

Getting a solution up and running with BenchBot is as simple as 1,2,3. Here's how to use BenchBot with content from the semantic scene understanding add-on:

  1. Run a simulator with the BenchBot software stack by selecting an available robot, environment, and task definition:

    u@pc:~$ benchbot_run --robot carter_omni --env miniroom:1 --task semantic_slam:active:ground_truth

    A number of useful flags exist to help you explore what content is available in your installation (see --help for full details). For example, you can list what tasks are available via --list-tasks and view the task specification via --show-task TASK_NAME.

  2. Create a solution to a BenchBot task, and run it against the software stack. To run a solution you must select a mode. For example, if you've created a solution in my_solution.py that you would like to run natively:

    u@pc:~$ benchbot_submit --native python my_solution.py

    See --help for other options. You also have access to all of the examples available in your installation. For instance, you can run the hello_active example in containerised mode via:

    u@pc:~$ benchbot_submit --containerised --example hello_active

    See --list-examples and --show-example EXAMPLE_NAME for full details on what's available out of the box.

  3. Evaluate the performance of your system using a supported evaluation method (see --list-methods). To use the omq evaluation method on my_results.json:

    u@pc:~$ benchbot_eval --method omq my_results.json

    You can also simply run evaluation automatically after your submission completes:

    u@pc:~$ benchbot_submit --evaluate-with omq --native --example hello_eval_semantic_slam

The BenchBot Tutorial is a great place to start working with BenchBot; the tutorial takes you from a blank system to a working Semantic SLAM solution, with many educational steps along the way. Also remember the examples in your installation (benchbot-addons/examples_base is a good starting point) which show how to get up and running with the BenchBot software stack.

Power tools for autonomous algorithm evaluation

Once you are confident your algorithm is a solution to the chosen task, the BenchBot software stack's power tools allow you to comprehensively explore your algorithm's performance. You can autonomously run your algorithm over multiple environments, and evaluate it holistically to produce a single summary statistic of your algorithm's performance. Here are some examples again with content from the semantic scene understanding add-on:

Using BenchBot in your research

BenchBot was made to enable and assist the development of high quality, repeatable research results. We welcome any and all use of the BenchBot software stack in your research.

To use our system, we just ask that you cite our paper on the BenchBot system. This will help us follow uses of BenchBot in the research community, and understand how we can improve the system to help support future research results. Citation details are as follows:

@misc{talbot2020benchbot,
    title={BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots},
    author={Ben Talbot and David Hall and Haoyang Zhang and Suman Raj Bista and Rohan Smith and Feras Dayoub and Niko Sünderhauf},
    year={2020},
    eprint={2008.00635},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

If you use our benchbot environments for active robotics (BEAR) which are installed by default, we ask you please cite our data paper on BEAR. Citation details are as follows:

@article{hall2022bear,
author = {David Hall and Ben Talbot and Suman Raj Bista and Haoyang Zhang and Rohan Smith and Feras Dayoub and Niko Sünderhauf},
title ={BenchBot environments for active robotics (BEAR): Simulated data for active scene understanding research},
journal = {The International Journal of Robotics Research},
volume = {41},
number = {3},
pages = {259-269},
year = {2022},
doi = {10.1177/02783649211069404},
}

Components of the BenchBot software stack

The BenchBot software stack is split into a number of standalone components, each with their own GitHub repository and documentation. This repository glues them all together for you into a working system. The components of the stack are:

Further information

Supporters

Development of the BenchBot software stack was directly supported by:

QUT Centre for Robotics    CSIRO Machine Learning and Artificial Intelligence Future Science Platform (MLAI FSP)    



NVIDIA AI & Robotics    Australian Centre for Robotic Vision