If you only want to evaluate your summarization/translation model for yanolja, then the only thing you need is to follow this guide and get evaluation results.
To evaluate your model, you should install bleurt and BARTScore first.
pip install --upgrade pip # ensures that pip is current
git clone https://github.com/google-research/bleurt.git
pip install ./bleurt
The BARTScore repository below is pip installable version of BARTScore with MBart (for supporting korean and so on).
git clone https://github.com/Y-IAB/BARTScore
pip install -e ./BARTScore
After that, please install this repository.
git clone https://github.com/Y-IAB/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .[openai]
Before you use this repository, you need to set some environment variables. Please set your environment variables with your key and endpoints.
export AZURE_OPENAI_API_KEY="..."
export AZURE_API_VERSION="2023-12-01-preview"
export AZURE_ENDPOINT="....openai.azure.com"
export AVAILABLE_GPUS=0,1,2,3,4,5
Before you evaluate your models, you need to download dataset first. Please download dataset from the link and drop the JSONL dataset files into lm_eval/tasks/yanolja/data
.
After that, you can evaluate summarization/translation quality for your model:
lm_eval --model hf --model_args pretrained=yanolja/EEVE-Korean-Instruct-10.8B-v1.0 --tasks yanolja_summarization,yanolja_translation_ko-en,yanolja_translation_en-ko,yanolja_perplexity --limit 1000 --batch_size=auto
If you only want to evaluate summarization quality, then you can give only yanolja_summarization
for --tasks
argument.
lm_eval --model hf --model_args pretrained=yanolja/EEVE-Korean-Instruct-10.8B-v1.0 --tasks yanolja_summarization --limit 1000 --batch_size=auto
The table shows the task groups we prepared in advance and their descriptions:
Task Name | Description |
---|---|
yanolja_summarization(yasum , yasum_s ) |
This task group consists of various open-source summarization benchmarks (labeled) and real user input datasets from the yanolja reviewize service (unlabeled). The former are evaluated with reference-based metrics (BLEU, ROUGE, BLEURT, etc.), while the latter are evaluated with reference-free metrics (COMETKIWI, LLM Eval). |
yanolja_translation(yatrans , yatrans_s ) |
This task group consists of various open-source translation benchmarks (labeled) and real user input datasets from the yanolja babel service (unlabeled). The former are evaluated with reference-based metrics (BLEU, ROUGE, BLEURT, etc.), while the latter are evaluated with reference-free metrics (COMETKIWI, LLM Eval). Both direction (ko-en, en-ko) are supported. |
yanolja_translation_handmade(yatrans_hm , yatrans_hm_s ) |
This task group consists of human-labeled translations for travel domain. Because this task group is the most important for our task, we evaluate this group with all of metrics. Both direction (ko-en, en-ko) are supported. |
yanolja_perplexity | This task group measures the perplexity of LLM on yonolja real English/Korean data. By doing so, we determine the similarity between LLM's knowledge and yanolja data domain. |
All models supported in the original repository are available. Also, we add some translation API with our Fragma API service and Azure OpenAI Service. Below are three models that we expect to use frequently. | Model | Description | Model Args | Available Tasks | Example Usage |
---|---|---|---|---|---|
openai-chat-completions |
You can use OpenAI's various models (gpt-3.5-turbo, gpt-4-turbo, etc.) through this API. You can also use Azure OpenAI API by giving azure endpoint to base_url. You need to set AZURE_OPENAI_API_KEY environment variable. |
model, base_url | yanolja_translation, yanolja_summarization | lm_eval --model openai-chat-completions --model_args model=gpt-4-turbo,base_url=[BASE_URL] --tasks yanolja_summarization |
|
hf |
As in the original lm-evaluation-harness, you can evaluate the huggingface's model or a local model checkpoint. | pretrained | yanolja_translation, yanolja_summarization, yanolja_perplexity | lm_eval --model hf --model_args pretrained=yanolja/EEVE-Korean-Instruct-10.8B-v1.0 --tasks yanolja_summarization |
|
translator |
You can use various translators through Yanolja's Fragma API. See the documentation for the available translators. | api_key, target_lang, endpoint, model | yanolja_translation | lm_eval --model translator --tasks yanolja_translation --model_args api_key=[FRAGMA_API_KEY],target_lang=en,endpoint=[FRAGMA_ENDPOINT],model=[FRAGMA_MODEL] --task_config doc_to_text="{{source}}" |
By default, the template prepared for each task is set to our local model.
doc_to_text: "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful and concise answers to the user's questions.
Human: 주어진 한국어 문장을 영어로 번역해주세요. 번역된 문장은 영어로 시작해야 합니다.
Korean: {{source}}
Assistant: "
However, if you need to use the translate API, for example, you may need to change the template directly. In this case, you can change the prompt template in the command line argument as shown below.
lm_eval --model translator --tasks yanolja_translation --model_args api_key=[FRAGMA_API_KEY],target_lang=en,endpoint=https://fragma.prod.yanolja.in/pre/translate,model=deepl --task_config doc_to_text="{{source}}"
Before using Puree dataset, you need to configure GCP credential for you machine. There are several options to do it:
~/.config/gcloud/application_default_credentials.json
To use a Puree dataset on evaluation, you can pass Puree dataset URI on dataset_path
. The URI has puree://<dataset-id>
format, and you can find the dataset-id
from Puree dataset detail page URL.
# Original task with local dataset
task: mdn_en-ko
dataset_name: mdn_en-ko
dataset_path: json
dataset_kwargs:
data_files: ./data/MDN_translation_length_20_ko-en.jsonl
include: translation_template_en-ko.yaml
---
# Updated task with puree dataset
task: mdn_en-ko
dataset_name: mdn_en-ko
dataset_path: puree://ohgvC3elEgRFxvUXDkWe
include: translation_template_en-ko.yaml
LiteLLM allows you to evaluate models in the same way as OpenAI models, even for models that are not yet supported, such as Gemini and mistral-large.
A new v0.4.0 release of lm-evaluation-harness is available !
New updates and features include:
Please see our updated documentation pages in docs/
for more details.
Development will be continuing on the main
branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub, or in the EleutherAI discord!
This project provides a unified framework to test generative language models on a large number of different evaluation tasks.
Features:
The Language Model Evaluation Harness is the backend for 🤗 Hugging Face's popular Open LLM Leaderboard, has been used in hundreds of papers, and is used internally by dozens of organizations including NVIDIA, Cohere, BigScience, BigCode, Nous Research, and Mosaic ML.
To install the lm-eval
package from the github repository, run:
git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
We also provide a number of optional dependencies for extended functionality. A detailed table is available at the end of this document.
transformers
To evaluate a model hosted on the HuggingFace Hub (e.g. GPT-J-6B) on hellaswag
you can use the following command (this assumes you are using a CUDA-compatible GPU):
lm_eval --model hf \
--model_args pretrained=EleutherAI/gpt-j-6B \
--tasks hellaswag \
--device cuda:0 \
--batch_size 8
Additional arguments can be provided to the model constructor using the --model_args
flag. Most notably, this supports the common practice of using the revisions
feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:
lm_eval --model hf \
--model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
--tasks lambada_openai,hellaswag \
--device cuda:0 \
--batch_size 8
Models that are loaded via both transformers.AutoModelForCausalLM
(autoregressive, decoder-only GPT style models) and transformers.AutoModelForSeq2SeqLM
(such as encoder-decoder models like T5) in Huggingface are supported.
Batch size selection can be automated by setting the --batch_size
flag to auto
. This will perform automatic detection of the largest batch size that will fit on your device. On tasks where there is a large difference between the longest and shortest example, it can be helpful to periodically recompute the largest batch size, to gain a further speedup. To do this, append :N
to above flag to automatically recompute the largest batch size N
times. For example, to recompute the batch size 4 times, the command would be:
lm_eval --model hf \
--model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
--tasks lambada_openai,hellaswag \
--device cuda:0 \
--batch_size auto:4
The full list of supported arguments are provided here, and on the terminal by calling lm_eval -h
. Alternatively, you can use lm-eval
instead of lm_eval
.
[!Note] Just like you can provide a local path to
transformers.AutoModel
, you can also provide a local path tolm_eval
via--model_args pretrained=/path/to/model
accelerate
We support two main ways of using Hugging Face's accelerate 🚀 library for multi-GPU evaluation.
To perform data-parallel evaluation (where each GPU loads a separate full copy of the model), we leverage the accelerate
launcher as follows:
accelerate launch -m lm_eval --model hf \
--tasks lambada_openai,arc_easy \
--batch_size 16
(or via accelerate launch --no-python lm_eval
).
For cases where your model can fit on a single GPU, this allows you to evaluate on K GPUs K times faster than on one.
WARNING: This setup does not work with FSDP model sharding, so in accelerate config
FSDP must be disabled, or the NO_SHARD FSDP option must be used.
The second way of using accelerate
for multi-GPU evaluation is when your model is too large to fit on a single GPU.
In this setting, run the library outside of the accelerate
launcher, but passing parallelize=True
to --model_args
as follows:
lm_eval --model hf \
--tasks lambada_openai,arc_easy \
--model_args parallelize=True \
--batch_size 16
This means that your model's weights will be split across all available GPUs.
For more advanced users or even larger models, we allow for the following arguments when parallelize=True
as well:
device_map_option
: How to split model weights across available GPUs. defaults to "auto".max_memory_per_gpu
: the max GPU memory to use per GPU in loading the model.max_cpu_memory
: the max amount of CPU memory to use when offloading the model weights to RAM.offload_folder
: a folder where model weights will be offloaded to disk if needed.These two options (accelerate launch
and parallelize=True
) are mutually exclusive.
Note: we do not currently support multi-node evaluations natively, and advise using either an externally hosted server to run inference requests against, or creating a custom integration with your distributed framework as is done for the GPT-NeoX library.
vLLM
We also support vLLM for faster inference on supported model types, especially faster when splitting a model across multiple GPUs. For single-GPU or multi-GPU — tensor parallel, data parallel, or a combination of both — inference, for example:
lm_eval --model vllm \
--model_args pretrained={model_name},tensor_parallel_size={GPUs_per_model},dtype=auto,gpu_memory_utilization=0.8,data_parallel_size={model_replicas} \
--tasks lambada_openai \
--batch_size auto
For a full list of supported vLLM configurations, please reference our vLLM integration and the vLLM documentation.
vLLM occasionally differs in output from Huggingface. We treat Huggingface as the reference implementation, and provide a script for checking the validity of vllm results against HF.
Our library also supports the evaluation of models served via several commercial APIs, and we hope to implement support for the most commonly used performant local/self-hosted inference servers.
To call a hosted model, use:
export OPENAI_API_KEY=YOUR_KEY_HERE
lm_eval --model openai-completions \
--model_args model=davinci \
--tasks lambada_openai,hellaswag
We also support using your own local inference server with servers that mirror the OpenAI Completions and ChatCompletions APIs.
lm_eval --model local-chat-completions --tasks gsm8k --model_args model=facebook/opt-125m,base_url=http://{yourip}:8000/v1
Note that for externally hosted models, configs such as --device
and --batch_size
should not be used and do not function. Just like you can use --model_args
to pass arbitrary arguments to the model constructor for local models, you can use it to pass arbitrary arguments to the model API for hosted models. See the documentation of the hosting service for information on what arguments they support.
API or Inference Server | Implemented? | --model <xxx> name |
Models supported: | Request Types: | ||
---|---|---|---|---|---|---|
OpenAI Completions | :heavy_check_mark: | openai-completions , local-completions |
All OpenAI Completions API models | generate_until , loglikelihood , loglikelihood_rolling |
||
OpenAI ChatCompletions | :heavy_check_mark: | openai-chat-completions , local-chat-completions |
All ChatCompletions API models | generate_until (no logprobs) |
||
Anthropic | :heavy_check_mark: | anthropic |
Supported Anthropic Engines | generate_until (no logprobs) |
||
Textsynth | :heavy_check_mark: | textsynth |
All supported engines | generate_until , loglikelihood , loglikelihood_rolling |
||
Cohere | :hourglass: - blocked on Cohere API bug | N/A | All cohere.generate() engines |
generate_until , loglikelihood , loglikelihood_rolling |
||
Llama.cpp (via llama-cpp-python) | :heavy_check_mark: | gguf , ggml |
All models supported by llama.cpp | generate_until , loglikelihood , (perplexity evaluation not yet implemented) |
||
vLLM | :heavy_check_mark: | vllm |
Most HF Causal Language Models | generate_until , loglikelihood , loglikelihood_rolling |
||
Mamba | :heavy_check_mark: | mamba_ssm |
Mamba architecture Language Models via the mamba_ssm package |
generate_until , loglikelihood , loglikelihood_rolling |
||
Huggingface Optimum (Causal LMs) | ✔️ | openvino |
Any decoder-only AutoModelForCausalLM converted with Huggingface Optimum into OpenVINO™ Intermediate Representation (IR) format | generate_until , loglikelihood , loglikelihood_rolling |
... | |
Neuron via AWS Inf2 (Causal LMs) | ✔️ | neuronx |
Any decoder-only AutoModelForCausalLM supported to run on huggingface-ami image for inferentia2 | generate_until , loglikelihood , loglikelihood_rolling |
... | |
Your local inference server! | :heavy_check_mark: | local-completions or local-chat-completions (using openai-chat-completions model type) |
Any server address that accepts GET requests using HF models and mirror's OpenAI's Completions or ChatCompletions interface | generate_until |
... |
Models which do not supply logits or logprobs can be used with tasks of type generate_until
only, while local models, or APIs that supply logprobs/logits of their prompts, can be run on all task types: generate_until
, loglikelihood
, loglikelihood_rolling
, and multiple_choice
.
For more information on the different task output_types
and model request types, see our documentation.
A number of other libraries contain scripts for calling the eval harness through their library. These include GPT-NeoX, Megatron-DeepSpeed, and mesh-transformer-jax.
To create your own custom integration you can follow instructions from this tutorial.
[!Note] For tasks unsuitable for direct evaluation — either due risks associated with executing untrusted code or complexities in the evaluation process — the
--predict_only
flag is available to obtain decoded generations for post-hoc evaluation.
If you have a Metal compatible Mac, you can run the eval harness using the MPS back-end by replacing --device cuda:0
with --device mps
(requires PyTorch version 2.1 or higher).
[!Note] You can inspect what the LM inputs look like by running the following command:
python write_out.py \ --tasks <task1,task2,...> \ --num_fewshot 5 \ --num_examples 10 \ --output_base_path /path/to/output/folder
This will write out one text file for each task.
To verify the data integrity of the tasks you're performing in addition to running the tasks themselves, you can use the --check_integrity
flag:
lm_eval --model openai \
--model_args engine=davinci \
--tasks lambada_openai,hellaswag \
--check_integrity
For models loaded with the HuggingFace transformers
library, any arguments provided via --model_args
get passed to the relevant constructor directly. This means that anything you can do with AutoModel
can be done with our library. For example, you can pass a local path via pretrained=
or use models finetuned with PEFT by taking the call you would run to evaluate the base model and add ,peft=PATH
to the model_args
argument:
lm_eval --model hf \
--model_args pretrained=EleutherAI/gpt-j-6b,parallelize=True,load_in_4bit=True,peft=nomic-ai/gpt4all-j-lora \
--tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
--device cuda:0
GPTQ quantized models can be loaded by specifying their file names in ,autogptq=NAME
(or ,autogptq=True
for default names) in the model_args
argument:
lm_eval --model hf \
--model_args pretrained=model-name-or-path,autogptq=model.safetensors,gptq_use_triton=True \
--tasks hellaswag
We support wildcards in task names, for example you can run all of the machine-translated lambada tasks via --task lambada_openai_mt_*
.
To save evaluation results provide an --output_path
. We also support logging model responses with the --log_samples
flag for post-hoc analysis.
Additionally, one can provide a directory with --use_cache
to cache the results of prior runs. This allows you to avoid repeated execution of the same (model, task) pairs for re-scoring.
For a full list of supported arguments, check out the interface guide in our documentation!
[!Tip] Running lm-evaluation-harness as an external library and can't find (almost) any tasks available? Run
lm_eval.tasks.initialize_tasks()
to load the library's stock tasks before callinglm_eval.evaluate()
orlm_eval.simple_evaluate()
!
You can seamlessly visualize and analyze the results of your evaluation harness runs using both Weights & Biases (W&B) and Zeno.
You can use Zeno to visualize the results of your eval harness runs.
First, head to hub.zenoml.com to create an account and get an API key on your account page. Add this key as an environment variable:
export ZENO_API_KEY=[your api key]
You'll also need to install the lm_eval[zeno]
package extra.
To visualize the results, run the eval harness with the log_samples
and output_path
flags.
We expect output_path
to contain multiple folders that represent individual model names.
You can thus run your evaluation on any number of tasks and models and upload all of the results as projects on Zeno.
lm_eval \
--model hf \
--model_args pretrained=EleutherAI/gpt-j-6B \
--tasks hellaswag \
--device cuda:0 \
--batch_size 8 \
--log_samples \
--output_path output/gpt-j-6B
Then, you can upload the resulting data using the zeno_visualize
script:
python scripts/zeno_visualize.py \
--data_path output \
--project_name "Eleuther Project"
This will use all subfolders in data_path
as different models and upload all tasks within these model folders to Zeno.
If you run the eval harness on multiple tasks, the project_name
will be used as a prefix and one project will be created per task.
You can find an example of this workflow in examples/visualize-zeno.ipynb.
With the Weights and Biases integration, you can now spend more time extracting deeper insights into your evaluation results. The integration is designed to streamline the process of logging and visualizing experiment results using the Weights & Biases (W&B) platform.
The integration provide functionalities
results.json
file as an artifact for version control,<task_name>_eval_samples.json
file if the samples are logged,First you'll need to install the lm_eval[wandb] package extra. Do pip install lm_eval[wandb]
.
Authenticate your machine with an your unique W&B token. Visit https://wandb.ai/authorize to get one. Do wandb login
in your command line terminal.
Run eval harness as usual with a wandb_args
flag. Use this flag to provide arguments for initializing a wandb run (wandb.init) as comma separated string arguments.
lm_eval \
--model hf \
--model_args pretrained=microsoft/phi-2,trust_remote_code=True \
--tasks hellaswag,mmlu_abstract_algebra \
--device cuda:0 \
--batch_size 8 \
--output_path output/phi-2 \
--limit 10 \
--wandb_args project=lm-eval-harness-integration \
--log_samples
In the stdout, you will find the link to the W&B run page as well as link to the generated report. You can find an example of this workflow in examples/visualize-wandb.ipynb, and an example of how to integrate it beyond the CLI.
For more information on the library and how everything fits together, check out all of our documentation pages! We plan to post a larger roadmap of desired + planned library improvements soon, with more information on how contributors can help.
To implement a new task in the eval harness, see this guide.
In general, we follow this priority list for addressing concerns about prompting and other eval details:
These are guidelines and not rules, and can be overruled in special circumstances.
We try to prioritize agreement with the procedures used by other groups to decrease the harm when people inevitably compare runs across different papers despite our discouragement of the practice. Historically, we also prioritized the implementation from Language Models are Few Shot Learners as our original goal was specifically to compare results with that paper.
The best way to get support is to open an issue on this repo or join the EleutherAI Discord server. The #lm-thunderdome
channel is dedicated to developing this project and the #release-discussion
channel is for receiving support for our releases. If you've used the library and have had a positive (or negative) experience, we'd love to hear from you!
Extras dependencies can be installed via pip install -e ".[NAME]"
Name | Use |
---|---|
anthropic | For using Anthropic's models |
dev | For linting PRs and contributions |
gptq | For loading models with GPTQ |
hf_transfer | For speeding up HF Hub file downloads |
ifeval | For running the IFEval task |
neuronx | For running on AWS inf2 instances |
mamba | For loading Mamba SSM models |
math | For running math task answer checking |
multilingual | For multilingual tokenizers |
openai | For using OpenAI's models |
optimum | For running Intel OpenVINO models |
promptsource | For using PromptSource prompts |
sentencepiece | For using the sentencepiece tokenizer |
testing | For running library test suite |
vllm | For loading models with vLLM |
zeno | For visualizing results with Zeno |
--------------- | --------------------------------------- |
all | Loads all extras (not recommended) |
@misc{eval-harness,
author = {Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy},
title = {A framework for few-shot language model evaluation},
month = 12,
year = 2023,
publisher = {Zenodo},
version = {v0.4.0},
doi = {10.5281/zenodo.10256836},
url = {https://zenodo.org/records/10256836}
}