AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

What's New

• [2024.01.15] 📣 AgentBoard is released.
• [2024.03.11] 🥳 AgentBoard is accepted by LLMAgents @ ICLR 2024

Introduction

AgentBoard emphasizes analytical evaluation for Large Language Models (LLMs) as generalist agents to perceive and act within various environments. It outlines four principles for constructing a benchmark to evaluate LLMs as generalist agents:

1. Task Diversity: AgentBoard incorporates 9 distinct tasks to comprehensively understand the generalist ability of LLM agents, which is built upon LLM's extensive knowledge base and exceptional scenario comprehension.
2. Multi-round Intercation: AgentBoard provides multi-round interaction between agents and environment, which is necessary to reflect the evolutionary nature of human intelligence, which continuously receives information and adapts towards the environment.
3. Partially-Observable Environments: In AgentBoard, the complete state of the environment is not available to the agent, which assesses agent world modeling ability as additional knowledge needs to be acquired through online exploration.
4. Analytical Evaluation: AgentBoard is a systematic evaluation platform: it includes a user-friendly script to construct goal-oriented reflex agents for a range of models, and features a panel for visualizing and interpreting results across multiple dimensions of agent proficiency, including fine-grained progress rates, grounding accuracy, performance breakdown for hard and easy examples, long-range in- teractions, detailed performance across various sub-skills, and trajectory with friendly visualization

Table of Contents

🚀 Quick Start

Here we provide a quick start guide to evaluate LLM agents on AgentBoard within 30 minutes.

Setup Environment

We provide both local setup (recommended) and docker as follows:

Setup Environment Variables in AgentBoard/.env

Environment Variables needed for AgentBoard include:

PROJECT_PATH = {path to project}/AgentBoard

ANTHROPIC_API_KEY=...
OPENAI_API_KEY=...

TODO_KEY=...
MOVIE_KEY=...
SHEET_EMAIL=...

WANDB_API_KEY=...

Evaluate Models

Example script for GPT-3.5-Turbo:

python agentboard/eval_main.py \
    --cfg-path eval_configs/main_results_all_tasks.yaml \
    --tasks alfworld \
    --model gpt-3.5-turbo-0613 \
    --wandb \
    --log_path ./results/gpt-3.5-turbo-0613 \
    --project_name evaluate-gpt-35-turbo-0613 \
    --baseline_dir ./data/baseline_results

We now offer configuration for 12 SOTA LLM models (gpt-4,gpt-3.5-turbo-0613, text-davinci-003,claude2,deepseek-67b,lemur-70b, mistral-7b,codellama-13b(34b),llama2-13b(70b),vicuna-13b-16k) and a simple reflex agent based on act-only prompting. You could also customize your own agents and LLMs. Models supported by vLLM should be generally supported in AgentBoard, while different models may require specific prompt templates.

Launch AgentBoard Analytical Evaluation Panel

AgentBoard integrates illustrative Weights&Bias visualization to help researchers better systematically analyze LLM agents. You can simply turn on --wandb switch in the arguments and customize the project_name and baseline_dir of your wandb project as the evaluation command above.

Before running, you need to setup wandb login or environment variable as instructed in quick-start. The visualization results would be both stored offline at \wandb. Normally after executing the evaluation command, you can visualize the live AgentBoard panel online at https://wandb.ai/{your_wandb_id}/{project_name}. We provide example WandB logging pages for GPT-4, GPT-3.5-Turbo, and DeepSeek-67b.

Note that if your run is not logged online (on a cluster without internet), you could later sync local runs to wandb online with wandb sync [OPTIONS] [PATH].. as detailed in wandb docs. For more information about the features of the AgentBoard panel, Please kindly check this Blog for more information.

Local log files

In addition to online results viewing, local logs are automatically stored in {log_path}. In WebArena, we additionally support more detailed trajectory files, including web page screenshots and network traffic records.

Data

Data Overview

AgentBoard is composed of 9 diverse tasks which can be divided into 4 types, including Embodied AI, Game, Web, and Tool:

To help researchers quickly understand evaluation data of each task, we provide Dataset Viewer at Huggingface Dataset: 🤗 AgentBoard.

Note: Please download the dataset from the link provided below for the reason that the data in Dataset Viewer is not complete.

Download Link

You can download the whole evaluation data by running the following command:

wget https://huggingface.co/datasets/hkust-nlp/agentboard/resolve/main/data.tar.gz

Please uncommpress the file and move the data to AgentBoard/data.

cd AgentBoard
mkdir data
tar -zxvf data.tar.gz

The file structure of evaluation data is as follows:

Evaluation Details

Evaluation Preparation

Internet Access

For regions with Internet restrictions, to evaluate the Tool-Query, Tool-Operation and WebArena tasks, please make sure that the machine can access the Internet.

You can check whether you have network issues by observing the output during the execution process.

Environment Preparation

We provide two ways to install the environment of AgentBoard, as specified in QuickStart.

Running Proprietary Models

In this section, we provide a script to evaluate the closed-source models on each task.

Please do not forget to set the environment variables (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY) before running the following commands.

For Tasks except WebShop

We provide a quick start script to evaluate the gpt-3.5-turbo-0613 model on alfworld task.

python agentboard/eval_main.py \
    --cfg-path eval_configs/main_results_all_tasks.yaml \
    --tasks alfworld \
    --model gpt-3.5-turbo-0613 \
    --wandb \
    --log_path ./results/gpt-3.5-turbo-0613 \
    --project_name evaluate-gpt-35-turbo-0613 \
    --baseline_dir ./data/baseline_results

Parameters:

• --cfg-path: The path of the config file, please refer to eval_configs/main_results_all_tasks.yaml for more details.
• --tasks: The tasks to be evaluated, e.g. tool-query, tool-operation, webarena, alfworld, babyai, jericho, pddl, scienceworld.
• --model: The LLM to be evaluated. We provide some LLM models, including:
  • gpt-3.5-turbo
  • gpt-3.5-turbo-16k
  • gpt-4
  • text-davinci-003
  • claude2
• wandb: Online visualization will be launched given this parameter. Remove this parameter from the script if you don't need visualization, e.g. during debugging.
• log_path: Path to save logs, as specified here.
• project_name: Project name for Weights&Bias. This parameter is not necessary when wandb parameter is not used.
• baseline_dir: Directory to results files of baseline models you want to compare with during the run.

For WebShop

First, please start the WebShop server by running the following commands:

cd ./agentboard/environment/WebShop
bash ./run_dev.sh
cd ../../..

Then, run the following command to evaluate the gpt-3.5-turbo-0613 model on webshop task.

python agentboard/eval_main.py \
    --cfg-path eval_configs/main_results_all_tasks.yaml \
    --tasks webshop \
    --model gpt-3.5-turbo-0613 \
    --wandb \
    --log_path ./results/gpt-3.5-turbo-0613 \
    --project_name evaluate-gpt-35-turbo-0613 \
    --baseline_dir ./data/baseline_results

Running Open-source Models

In AgentBoard, we have pre-supported the following 8 open-source models, by default we use vLLM to speed up inference.

• llama2-13b
• llama2-34b
• codellama-13b
• codellama-34b
• vicuna-13b-16k
• lemur-70b
• deepseek-67b
• mistral-7b

Please refer to eval_configs/main_results_all_tasks.yaml for more details about these models.

To evaluate these models, you can run the following command:

python agentboard/eval_main.py \
    --cfg-path eval_configs/main_results_all_tasks.yaml \
    --tasks ${TASK_NAME} \
    --model ${OPEN_SOURCE_MODEL_NAME}

We also provide LLM customizations, please refer to LLM Customization for more details.

LLM Customization

Please refer to llm_customization.md for more details about LLM customization.

Agent Customization

Please refer to agent_customization.md for more details about agent customization.

Runtime Estimation

The evaluation runtime for a language model depends on the device/API, model, and inference architecture used. In the case of open-source LLMs, the vllm inference speed is approximately 10 times faster than the huggingface pipeline.

To estimate the total time needed for evaluation, you can run a few steps to measure the inference speed and multiply it by the total number of LLM inferences, which is within 15,000 rounds.

The general formula for estimating the total time is 4h * speed. Here are some examples of our runtime:

️Citation

If you find this repository useful, please consider giving star and citing our paper:

@misc{ma2024agentboard,
      title={AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents}, 
      author={Chang Ma and Junlei Zhang and Zhihao Zhu and Cheng Yang and Yujiu Yang and Yaohui Jin and Zhenzhong Lan and Lingpeng Kong and Junxian He},
      year={2024},
      eprint={2401.13178},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

The AgentBoard codebase is licensed under a Apache-2.0 License.

The AgentBoard dataset is licensed under a GNU General Public License, version 2.