Contributing

We are accepting PRs for new models, evaluators, and eval sets, in addition to bug fixes. We will update the leaderboard website regularly with new community contributions. We have also created a support discord for AlpacaEval in case you run into any issues and wish to ask help from the community.

To get started, please first fork the repo, and install the package from source pip install -e .

Contributing a model

First, you'll need to add a model config definition in the models_configs folder. As an example, you can look at the falcon-7b-instruct yaml. Please make sure the folder name and key name in the yaml match exactly.

Then, please follow the steps in Evaluating a model to run inference on the model to produce outputs on the eval set and score the model according to one of the evaluators. An example command may look like:

alpaca_eval evaluate_from_model \
  --model_configs 'falcon-7b-instruct'

After running this command, you should have generated an outputs json and a new entry in the corresponding leaderboard file. Please make a PR with the config, outputs file, and updated leaderboard.

Concretely you should do something like:

1. Fork the repository in github
2. Clone the forked repository git clone <URL>
3. Make a model config at src/alpaca_eval/models_configs/<model_name> and evaluate it evaluate_from_model --model_configs '<model_name>'
4. Add the model configs, output, and leaderboard entry to the forked repository

   git add src/alpaca_eval/models_configs/<model_name> # add the model config
   git add src/alpaca_eval/leaderboards/ # add the actual leaderboard entry
   git add src/alpaca_eval/metrics/weights # add the weights for LC
   git add -f results/<model_name>/model_outputs.json # force add the outputs on the dataset
   git add -f results/<model_name>/*/annotations.json # force add the evaluations from the annotators
   git commit -m "Add <model_name> to AlpacaEval"
   git push

5. Create a pull request on AlpacaEval

Note: if you are generating outputs outside of AlpacaEval you should still add a model config but with fn_completions: null. See this config for an example.

Limitations

The AlpacaEval evaluation pipeline, like other current evaluators have important limitations and should therefore not be used as replacement for human evaluation in important settings, such as to decide whether a model is ready to be deployed. Those can broadly be clustered into 3 categories:

1. Instructions might not be representative of real-usage: the AlpacaEval set contains examples from a variety of datasets (self-instruct, open-assistant, vicuna, koala, hh-rlhf) which might not be representative of real-usage and advanced applications of better models like GPT4. This likely makes the best closed models (GPT4 / Claude / ChatGPT / ...) seem more similar to the open models than what they are. Indeed, those closed models seem to be pretrained/finetuned on much more diverse data. See for example this blog for preliminary results on more complex instructions. Note, however, that in AlpacaFarm we showed that win-rates on our evaluation set are highly correlated (0.97 R2) with win-rates on instructions from user interactions with the Alpaca Demo. Furthermore, the AlpacaEval leaderboard shows larger gap between the open models and OpenAI models than other leaderboards ( e.g. lmsys).

2. Biases of automatic annotators: the raw automatic annotators seem to have implicit biases. In particular, we found that they tend to prefer longer outputs and outputs that contain lists (e.g. 0.68 / 0.69 for alpaca_eval_gpt4 and 0.62 / 0.58 for claude). Although we found that humans have similar biases (0.64 / 0.61), we believe that this could be more of a limitation of human annotation pipeline we used rather than a true human bias. More generally, through qualitative analysis, we found that automatic annotators give more importance to the style of the output than its content (e.g. factuality). Finally, we found that automatic evaluators tend to prefer outputs from models that are similar (likely trained on the same data) as suggested by the big difference between ChatGPT/GPT4 on claude's and alpaca_eval_gpt4's leaderboard. Note that the length bias is partially mitigated in our length-controlled win-rates.

3. Lack of safety evaluation: importantly, AlpacaEval only evaluates the instruction-following capabilities of models rather than the harm that they could cause (e.g. toxic behavior or bias). As a result the small gap between current ChatGPT and the best open source models should not be interpreted as if that the latter are ready to be deployed.

Beyond those limitations about the evaluation pipelines, there are also limitations about our validation of the evaluators and our proposed approach to selecting evaluation sets.

Additional analysis and plots

Length-controlled AlpacaEval (LCAE)

Length-controlled AlpacaEval Visualizations:

Length-controlled AlpacaEval Development:

The notebook shows different options that we considered for mitigating the length bias of automatic annotators.

Here we briefly summarize the main results. Namely:

• LCAE increases the correlation with Chat Arena to 0.98 from 0.94 for AlpacaEval 2.0. This makes LCAE the most highly correlated benchmark with Chat Arena as seen in the plot below.

• LCAE decreases length gameability one of the major issues of AlpacaEval is that you can increase your win-rate by increasing the length of your outputs. For example, in AlpacaEval 2.0 the win-rate for the baseline (50%) increases to 64% when prompted to “give as much detail as possible” and decreases to 23% when prompted to “be as concise as possible while still providing all the necessary information to answer the question”. More generally the relative length gameability was ~21% for AlpacaEval and decreases to ~6% for LCAE, so it's 3x less gameable through prompt length. This is shown in the plot below.

• We can predict performance for different baselines One other benefit of using a GLM for controlling for length bias. Is that we now have a model that can predict the win-rate of a model for different baselines. In particular, our GLM has many nice properties, for example win_rate(m,b) = 1 - win_rate(b,m) \in [0,1] and win_rate(m,m) = 0.5. This is shown in the plot below.

Finally, note that we are only controlling for length bias. There are other known biases that we are not controlling for, such as the fact that auto-annotators prefer outputs similar to their model. Although we could control for that, in practice we have found that to be less of an issue than length bias. For two reasons (1) this mostly a single model in the leaderboard because fine-tuning on outputs from the auto-annotator doesn't seem to have doesn't seem to impact the win-rate as much, and (2) the bias is actually less strong that what one could think. For example we show below a subset of the leaderboards auto-annotated by three different models, and we see that the ranking of models is exactly the same. In particular, claude-3-opus prefers gpt4_preview, and mistral-large prefers the former two.

Citation

Please consider citing the following depending on what you are using and referring to:

• Code, results, and general benchmark: alpaca_eval (this repo). Specify whether you are using AlpacaEval or AlpacaEval 2.0. For length-controlled win-rates see below.
• Length-controlled (LC) win rates: alpaca_eval_length.
• Human annotations: dubois2023alpacafarm (AlpacaFarm)
• AlpacaEval evaluation set: alpaca_eval and self-instruct, open-assistant, vicuna, koala, hh-rlhf.

Here are the bibtex entries:

@misc{alpaca_eval,
  author = {Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {AlpacaEval: An Automatic Evaluator of Instruction-following Models},
  year = {2023},
  month = {5},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/alpaca_eval}}
}

@article{dubois2024length,
  title={Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators},
  author={Dubois, Yann and Galambosi, Bal{\'a}zs and Liang, Percy and Hashimoto, Tatsunori B},
  journal={arXiv preprint arXiv:2404.04475},
  year={2024}
}

@misc{dubois2023alpacafarm,
  title={AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback}, 
  author={Yann Dubois and Xuechen Li and Rohan Taori and Tianyi Zhang and Ishaan Gulrajani and Jimmy Ba and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto},
  year={2023},
  eprint={2305.14387},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}

More information