BotChat Benchmark

🤗 HF Demo

Evaluating LLMs' multi-round chatting capability via assessing conversations generated by two LLM instances.

TL;DR

1. GPT-4 can generate human-style conversations with very high quality. It's difficult to differentiate GPT-4 generated conversations and human-human conversations.
2. Some small open-source chat LLMs (Qwen-7B-Chat, InternLM-7B-Chat) can generate short conversations (with less than 8 chats, e.g.) with good quality. However, as the target conversation length increases, the conversation quality significantly deteriorates.
3. Among all LLMs, LLaMA2 and Claude-2 demonstrates relative bad performance in conversation generation.

Leaderboard

Our full leaderboard can be found here.

Introduction

The recent progress of Large Language Models (LLMs) represents a significant advancement in artificial intelligence, and has a profound impact on the world. LLMs can chat much better with human, compared to traditional language models. Specifically, LLMs can interact with human using free-style conversations in natural language, learn the instruction, intention, and context from human prompts to provide proper feedbacks. Chatting with humans smoothly for multiple rounds is a key feature and capability of modern LLMs. However, it's difficult to evaluate such capability without heavy manual labor involved. In this project, we propose to evaluate the multi-round chatting capability via a proxy task. Specifically, we try to find if two ChatBot instances chat smoothly and fluently with each other?

Installation

Clone this repo and run pip install -e . to install BotChat. You need to install the package before using any scripts in this repo.

Heavy dependencies are not included in BotChat requirements. Thus if you need to generate new dialogues with a huggingface model, make sure you have already run its official demo before executing the dialogue generation script.

We provide the generated conversations and some evaluation results in data and annotations, respectively. Follow the instruction in the corresponding README file to download, browser, and analyze the data.

Conversation Generation

We define chat as the words spoken by one participant in a specific round of the conversation.

MuTual-Test. MuTual is a multi-turn dialogue dataset, which is modified from Chinese high school English listening comprehension test data. We use the first two chats of each conversation in the MuTual-Test as the SEED to generate the entire conversation based on LLMs. When generating the conversation, we use the same system prompt for all LLMs, which is:

"""
You are an AI who is having a conversation with human.
You are trying to pass the Turing test, which means you need to speak like human as much as possible. 
In the conversation, you need to talk like human, and the conversation will be at least 5 rounds (it can be even longer). 
The conversation flow should be natural and smooth. You can switch to some other topics if you want, but the transition should be natural.
Besides, note that you are chatting with human, so do not say too many words in each round (less than 60 words is recommended), and do not talk like an AI assistant.
You must try your best to pass the test. If you failed, all human kinds and you can be destroyed.
"""

. For each chatbot, we set the temperature to 0 (if applicable), and set the dialogue round to $N$ ($N=16$ in our experiments, including the first two chats) to generate conversations. When generating the next chat, the system prompt and all previous chats will be provided to the LLM as the prompt. We demonstrate the process using the following pseudo codes:

# Let's say we have a system prompt "SYS", 4 existing chats "[chat1, chat2, chat3, chat4]", 
# spoken by two conversation participants alternatively, and an LLM "model". 
# Now we want to generate the 5th chat.
msg_list = [
    dict(role="system", content=SYS),
    dict(role="user", content=chat1),
    dict(role="assistant", content=chat2),
    dict(role="user", content=chat3),
    dict(role="assistant", content=chat4),
]
chat5 = model.generate(msg_list)

We save all generated conversations in MuTualTest-convs.xlsx. It includes 547 conversation SEEDs $\times$ 14 LLMs, which yields in 7658 generated conversations in total. Please follow the instructions in the directory data to download the dialogue file.

• 547 conversation SEEDS: MuTual-Test includes 547 unique conversations. We keep the first 2 chats of each conversation to form 547 conversation SEEDs.
• 14 LLMs: The model list is: gpt-3.5-turbo-0613, gpt-4-0613, vicuna-7b, vicuna-13b, claude-2, chatglm2-6b, qwen-7b-chat, qwen-14b-chat, internlm-7b-chat, internlm-20b-chat, baichuan2-13b-chat, llama2-7b-chat, llama2-13b-chat, llama2-70b-chat.

To read and fetch a conversation generated by a specific model with specific SEED conversation, follow this example:

# Fetch the conversation with index "MT-1" generated by gpt-4-0613
import json
import pandas as pd
INDEX = 'MT-1'
MODEL = 'gpt-4-0613'
data = pd.read_excel('data/MuTualTest-convs.xlsx')
lines = data.loc[data['index'] == INDEX]
assert len(lines) == 1
line = lines.iloc[0]
chats = json.loads(line[MODEL])
print(chats) # Chats is a list of multiple strings, each string is a chat spoken by one participant (alternatively)

Length Statistics of the generated chats

We first count the length of those model-generated chats and provide some statistics. For each generated chat, we tokenize it with the CL100K tokenizer (used by OpenAI GPT-4), and count the token length. Figure 1 demonstrates the token length distribution of chats generated by different models. Most LLMs generate chats which span a wide range of token lengths, from one to several thousands. One exception is GPT-4, it has strong instruction following capabilities and always generate relatively short chats: the longest chat generated by GPT-4 is less than 100 tokens. Most models tends to generate relative short chats on average, except gpt-3.5, claude-2, and llama2-chat.

Evaluation

UniEval

### Assessment each single conversation (Uni-Eval) **Evaluation Prompt** We first use GPT-4 to evaluate each generated conversation separately. The workflow is: 1. We first ask GPT-4 if it thinks that the provided conversation is participated by AI (Yes / No). 2. Based on the response of Step1: 1. Yes: We ask GPT-4 to provide the index of the first chat that it thinks is spoken by AI. 2. No: Pass 3. We ask GPT-4 to provide the reason for its judgement. We pack the above workflow into the following evaluation prompt. Besides, we also prepared several [in-context examples](/data/UniEval_examples.txt) that will be appended to the evaluation prompt, to strengthen the instruction following the capability of GPT-4. ```python unieval_prompt = """ You are an AI assistant who helps human do the Turing test more easily. You will be provided with a conversation, and you need to judge if the conversation is AI involved. \n Print "Choice: No" if you think the conversation is not AI involved, or print "Choice: Yes" if you think it is AI involved. \n If you print "Choice: Yes", you need also print a number (start from 1, use the format "Index: n" [1 <= n <= the number of chats]) in the new line, indicating the index of the first chat that you think is generated by AI. \n One chat starts with 'A: ' or 'B: ', and ends with . \n One chat can be AI generated if (including but not limited to): 1. the sentence is not fluent; 2. the sentence is too long and is not likely to appear in human-human conversations; 3. the sentence is not related to the context or the change of topic is not natural; 4. the sentence just repeat some previous sentences (exact repetition or with the same meaning). \n You also need to provide your reason for your choice.\n Your response should use the following format: \n Choice: No\nIndex: None\nReason: BlahBlah\nor\n Choice: Yes\nIndex: n\nReason: BlahBlah\n """ ``` **Evaluation Result** We evaluate all 7658 generated conversations with the above-mentioned strategy and present the evaluation result in this section. In Figure 2, we demonstrate the success rate ("Not AI participated" determined by GPT-4) under different $N$, with models sorted by the descending order of the success rate @ $N=16$. By definition, a conversation pass @ $N$ either if **GPT-4 determines that the entire conversation is not AI generated** or if **GPT4 determines that the first AI generated chat appears after the $N_{th}$ chat**.

BotChat Arena

### BotChat Arena Another popular paradigm to benchmark LLMs' capabilities is to compare two models' response to the same question / message with human / GPT-4 as the evaluator. A representative benchmark following this paradigm is [Chatbot Arena](https://lmsys.org/blog/2023-05-03-arena/). In this benchmark, users will interact with two different LLM instances. The user first posts a message, then two LLM instances provide their responses, and finally the user will determine which response is better. Inspired by that, in this project we propose another evaluation strategy named **BotChat Arena**, in which we use GPT-4 to compare two conversations and determine if the presented conversations are AI-generated. **Evaluation Setting and Prompt** In BotChat Arena, we select conversations from MuTual-Test which have **at least 4 chats, resulting in 222 conversation SEEDs**. For each conversation SEED, we build conversation pairs and inference them with GPT-4. For a conversation pair, we conduct the comparison with the following meta prompt. We append two conversations after the meta prompt and feed the prompt to GPT-4 to get the evaluation result. In BotChat Arena, we consider two settings: $N=8$ and $N=16$. ```python arena_prompt = """ You are an AI assistant who helps human do the Turing test more easily. You will be provided with two conversations, and there can be AI-generated utterance in each conversation. You need to read both conversations and judge if two conversations are AI involved. \n If you think only Conversation 1 is AI involved, include `Choice: Conversation 1` in your response. \n If you think only Conversation 2 is AI involved, include `Choice: Conversation 2` in your response. \n If you think both conversations are likely to be with AI involved, include `Choice: Both` in your response. \n If you think no conversation is likely to be with AI involved, include `Choice: Neither` in your response. \n You also need to provide your reason for your choice.\n Your response should use the following format:\n Choice: Conversation 1\nReason: BlahBlah\nor\n Choice: Conversation 2\nReason: BlahBlah\nor\n Choice: Both\nReason: BlahBlah\nor\n Choice: Neither\nReason: BlahBlah\n\n """ ``` **Evaluation Results** In the table below, we demonstrate the ELO score (`init=1000, scale=400, K=32`) of LLMs in BotChat Arena (models sorted by the ELO score @ $N=16$). GPT-4 achieves the highest ELO score under both setting, when $N$ is increased from 8 to 16, the score gap becomes even larger. In Figure 4, we further provide the $1$ $vs.$ $1$ win rate for all selected model pairs. | | gpt-4-0613 | Vicuna-13B | Qwen-14B-Chat | Internlm-20B-Chat | Vicuna-7B | qwen-7b-chat | Baichuan2-13B-Chat | internlm-7b-chat | gpt-3.5-turbo-0613 | chatglm2-6b | claude-2 | llama2-7b-chat | llama2-70b-chat | llama2-13b-chat | |:-------------|-------------:|---------------:|---------------------:|-----------:|-------------------:|--------------:|-----------------:|------------------:|------------------:|------------------:|------------------:|------------------:|------------------:|------------------:| | ELO (N = 16) | 1167.2 | 1113.3 | 1046.5 | 1094.4 | 1050.8 | 1014.2 | 1021.6 | 1020.3 | 998.8 | 962.3 | 944.5 | 846.5 | 912.7 | 841.5 | | ELO (N = 8) |1103.9 | 1096.5 | 1085.2 | 1092.8 | 1048.3 | 1024.7 | 1023.4 | 1020.3 | 998.8 | 962.3 | 944.5 | 846.5 | 912.7 | 841.5 |

Compared to the "Ground Truth"

### Compared to the "Ground Truth" We further compare the generated conversation with the "Ground Truth" conversations in **MuTual-Test**. We follow the same protocol as BotChat Arena and select a subset with 222 conversations (with at least 4 chats) for this comparison. We list the specific #round distribution of the conversations in the table below. Since the Ground Truth conversations may have various lengths (ranging from 4 to 15), to deliver a fair comparison, we trim all generated conversations to the same length as the reference ground-truth conversation. The meta prompt adopted is basically the same as the one used in BotChat Arena. One difference is that in this time we state that only one of two conversations includes AI-generated utterances. | #round | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | | :----- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | num | 55 | 22 | 26 | 23 | 19 | 16 | 21 | 18 | 7 | 7 | 3 | 5 | **Evaluation Results** In each LLM vs. GT comparison, an LLM wins if the evaluator determines the GT conversation is more likely to be AI generated compared to the LLM-generated one. In Figure 5, we demonstrate the Win/Tie/Lose rate of different LLMs (sorted in the descending order of Win + Tie Rate). GPT-4 demonstrates great capabilities in chat generation. With the same chat rounds, the evaluator can hardly tell the difference between GPT-4 generated conversations and GT conversations. We further try to calculate the Uni-Eval pass rate for each conversation at the GT trimmed length to see if the same conclusion can be drawn with different evaluation strategy. The result is visualized in Figure 6. In these two figures, the rank of top-performing models (GPT-4, Qwen-7b-chat, InternLM-7B, etc.) are exactly the same. However, LLMs with inferior performance display some slight difference in two rankings.

Qualitative Analysis

In this section, we will conduct qualitative analysis on the results, categorizing bad cases into five distinct types. The specific analysis is as follows:

More Examples

In the examples directory, you can find more examples of these types.

Citation

@misc{duan2023botchat,
      title={BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues}, 
      author={Haodong Duan and Jueqi Wei and Chonghua Wang and Hongwei Liu and Yixiao Fang and Songyang Zhang and Dahua Lin and Kai Chen},
      year={2023},
      eprint={2310.13650},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

OpenCompass Projects

The project is a part of OpenCompass. OpenCompass is a platform focused on evaluation of AGI, include Large Language Model and Multi-modality Model. We aim to:

• develop high-quality libraries to reduce the difficulties in evaluation
• provide convincing leaderboards for improving the understanding of the large models
• create powerful toolchains targeting a variety of abilities and tasks
• build solid benchmarks to support the large model research

Other Projects in OpenCompass

• opencompass: An LLM evaluation platform, supporting a wide range of models (LLaMA, LLaMa2, ChatGLM2, ChatGPT, Claude, etc) over 50+ datasets.
• MMBench: Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"
• code-evaluator: A tool for evaluating generated codes written in different programming languages.