This work presents a novel approach to generating behavior trees for robots using lightweight large language models (LLMs) with a maximum of 7 billion parameters. The study demonstrates that it is possible to achieve satisfying results with compact LLMs when fine-tuned on a specific dataset. The key contributions of this research include the creation of a finetuning dataset based on existing behavior trees using GPT-3.5 and a comprehensive comparison of multiple LLMs (namely llama2, llama-chat, and code-llama) across nine distinct tasks. To be thorough, we evaluated the generated behavior trees using static syntactical analysis, a validation system, a simulated environment, and a real robot. Furthermore, this work opens the possibility of deploying such solutions directly on the robot, enhancing its practical applicability. Findings from this study demonstrate the potential of LLMs with a limited number of parameters in generating effective and efficient robot behaviors.
Release for the paper BTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMs, currently in submission at IEEE/RSJ International Conference on Intelligent Robots and Systems.
Preprint available on arxiv: https://arxiv.org/abs/2403.12761
Dataset
, llama-2-7b-chat
and codellama-7b-instruct
LoRA adapters available on HuggingFace.
Authors: Riccardo Andrea Izzo, Gianluca Bardaro and Matteo Matteucci
Location: AIRLab (The Artificial Intelligence and Robotics Lab of Politecnico di Milano)
bt_client
: client designed to interpret and execute LLM-generated behavior trees directly on robotbt_generator
: demo notebook to load our fine-tuned models for generating behavior treesbt_validator
: validator that assess the overall correctness of the LLM-generated treesdataset
: our instruction-following dataset used to fine-tune the modelslora_adapters
: LoRA adapters for the base models, used in the notebook to load the fine-tuned versionprompt
: outcomes of prompts run on both LlamaChat and CodeLlama models, both in zero-shot and one-shot scenariosCreate a conda environment (or equivalent virtualenv):
conda create -n btgenbot python==3.10
Install dependencies:
pip install -r requirements.txt
Create a colcon workspace and clone this repository in your ROS2 workspace
Build:
colcon build
Required ROS2 dependencies:
BehaviorTree.CPP
: available hereBehaviorTree.ROS2
: available hereigus_rebel_commander
: available here, required only by bt_client
for the task involving arucos and arm activityaruco_interfaces
: available here, required only by bt_client
for the task involving arucos and arm activityTested on a Linux computer with Ubuntu 22.04 and ROS2 Humble
/bt_xml
config/tree.yaml
configuration file with the file name in the tree_name
field:
tree_name: "main_tree.xml"
To add a new behavior tree, follow these steps:
/bt_xml
folderconfig/tree.yaml
configuration filecolcon build
source install/setup.bash
ros2 launch bt_client bt.launch.py
Keep in mind that the system offers a range of pre-defined node functionalities. For instance, the "MoveTo" action facilitates the sending of a navigation goal to the Nav2 server, utilizing the goal specified within the behavior tree XML.
Moreover, locations for testing purposes are outlined in the location.yaml configuration file. These locations are pre-defined and serve as references for navigation tasks.
It is possible to add further actions with
factory.registerNodeType<ACTION_NAME>("ACTION_NAME");
ros2 launch bt_client monitor.launch.py
This command initiates the pipeline. Once the behavior tree specified in the config/tree.yaml
configuration file becomes available, the client will automatically execute it.
This behavior tree is intended to be the one generated by the LLM, for example with inference.ipynb
or btgenbot.py
.
Explore a demonstration notebook showcasing the generation of behavior trees utilizing llamachat and codellama, featuring both zero-shot and one-shot prompts.
Client application with GUI that generates a behavior tree given a new task description. After generating the behavior tree, the application saves it to a file and initiates its transmission to the remote location of a robot for immediate execution.
Two modes are available:
Standard Mode
: this mode requires a comprehensive one-shot example that includes a description and its corresponding behavior tree, in addition to the new task description. Recommended for new or specialized tasks.Automatic Retrieval Mode
: in this mode, only the new task description is required. Users can select the task domain to automatically infer a one-shot example from a list of predefined ones in a YAML file. Ideal for straightforward tasks similar to those demonstrated in the examples.Steps:
config/params.yaml
configuration file with SSH connection parameters and your HuggingFace access token. Define the local_dir
parameter to specify the local directory where XML behavior trees will be saved, and set remote_dir
to indicate the corresponding remote location on the robot where the trees should be stored.python3 btgenbot.py
colcon build
source install/setup.bash
./build/bt_validator/main
If you use this work in your research, please consider citing our paper:
@article{izzo2024btgenbot,
title={BTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMs},
author={Izzo, Riccardo Andrea and Bardaro, Gianluca and Matteucci, Matteo},
journal={arXiv preprint arXiv:2403.12761},
year={2024}
}