M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models

📃 Paper • 🤗 Dataset

📢 Update

[2024/07/27] We released the dataset that supports up to 128K. We also released the code for constructing the test instances. See 5.2. Data Creation for more details.
[2024/05/16] Our paper is accepted to ACL 2024 main.
[2023/10/31] We released the M4LE paper.

📚 Content

• 1. Introduction
• 2. Leaderboard
• 3. Results
• 4. Setup
• 5. Data
  • 5.1. Load Data
  • 5.2. Data Creation
• 6. Task
• 7. Inference
• 8. Evaluation
• Citation

📘 1. Introduction [Back to Top]

M4LE is a Multi-ability, Multi-range, Multi-task, bilingual benchmark for long-context evaluation. We categorize long-context understanding into five distinct abilities by considering whether it is required to identify single or multiple spans in long contexts based on explicit or semantic hints. Specifically, these abilities are explicit single-span, semantic single-span, explicit multiple-span, semantic multiple-span, and global. Different from previous long-context benchmarks that simply compile from a set of existing long NLP benchmarks, we introduce an automated method to transform short-sequence tasks into a comprehensive long-sequence scenario encompassing all these capabilities.

M4LE consists of 36 tasks, covering 11 task types and 12 domains. For each task, we construct 200 instances for each context length bucket (1K, 2K, 4K, 6K, 8K, 12K, 16K, 24K, 32K). Due to computation and cost constraints, our paper evaluated 11 well-established LLMs on instances up to the 8K context length bucket. For more details, please refer to the paper.

🏆 2. Leaderboard [Back to Top]

For each task, the score is normalized by the score of GPT-3.5-Turbo-16K on the 1K context. The equation for normalization can be found in the paper. We use the average normalized score obtained from context length buckets of 1K, 2K, 4K, 6K, and 8K to compare different models.

📊 3. Results [Back to Top]

🛠 4.️ Setup [Back to Top]

The following command sets up a conda environment for inference and evaluation.

conda env create --name m4le --file environment.yml

🗂️ 5. Data [Back to Top]

5.1. Load Data

As described in the paper, we adopt different open-sourced datasets to construct 36 tasks in M4LE. Each task has a jsonl file containing all the testing instances. You can load the data from the huggingface hub:

from datasets import load_dataset
tasks = [
    "arxiv",
    "bigpatent_global_cls",
    "bigpatent_global_sum",
    "booksum",
    "c3",
    "cepsum",
    "clts+",
    "cnewsum",
    "cnnnews",
    "drcd_explicit-single",
    "drcd_semantic-single",
    "duorc",
    "dureader",
    "hotpotqa",
    "lcsts",
    "marc",
    "mnds-news_explicit-single",
    "mnds-news_explicit-multiple",
    "mnds-news_semantic-multiple",
    "ncls",
    "news-commentary-en2zh",
    "news-commentary-zh2en",
    "news2016",
    "newsqa",
    "nq-open",
    "online-shopping",
    "open-subtitles-en2zh",
    "open-subtitles-zh2en",
    "pubmed",
    "tedtalks-en2zh",
    "tedtalks-zh2en",
    "thucnews_explicit-single",
    "thucnews_explicit-multiple",
    "thucnews_semantic-multiple",
    "triviaqa",
    "wiki2019zh",
    "wikihow",
    "wikitext-103",
    "wow",
]

for task in tasks:
    data = load_dataset('wckwan/M4LE', task, split='test')

Each testing instance follows this format:

{
    "instruction": "<task description>",
    "input": "<task input with one-shot example>",
    "answers": ["<answer1>", "<answer2>"],
    "input_length": <int, number of words in instruction and input separated by space>,
    "total_length": <int, number of words in instruction, input and gold answer separated by space>,
    "length_bucket": <int, the length bucket to which this instance belongs>
}

5.2. Data Creation

We also provide the scripts to construct the instances from the raw datasets.
First, download and preprocess the data. It downloads many source datasets which takes lots of time and space. It is possible to encounter network errors during the download process where you have to edit the script to resume where you left off.

./raw_data/preprocess.sh

Construct the test instances with the following script. To customize the length buckets, edit the variable buckets in create_data.sh

./data/create_data.sh

📝 6. Task [Back to Top]

🧠 7. Inference [Back to Top]

In the paper, we use the prompt format f"{instruction}\n{input}". We recommend editing or using our `inference.py`` code. To run the code, provide the HuggingFace model name and task name:

python inference.py \
--model_path lmsys/vicuna-13b-v1.5-16k \
--task nq-open

For LLaMA2 models, we recommend enabling dynamic NTK scaling:

python inference.py \
--model_path meta-llama/Llama-2-7b \
--task nq-open \
--load_model_args '{"rope_scaling": {"type": "dynamic", "factor": 2.0}}'

Additional arguments:

• --resume: Specify if you wish to pick up from where you left off.
• --api_key: The OpenAI key if you are using OpenAI models.

The predictions will be saved at outputs/{model}/{task}.jsonl

📈 8. Evaluation [Back to Top]

Run the following script to evaluate:

python evaluation.py

It will evaluate all the models saved at outputs/{model}/. The results will be saved at results/all_results.csv, where each row contains

• Model: Model name.
• Task: Task Name.
• Bucket: The context length bucket.
• Score: The evaluation score.

📄 Citation

If you find our paper and resources useful, please consider citing our paper:

@misc{kwan_m4le_2023,
  title = {{{M4LE}}: {{A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark}} for {{Large Language Models}}},
  author = {Kwan, Wai-Chung and Zeng, Xingshan and Wang, Yufei and Sun, Yusen and Li, Liangyou and Shang, Lifeng and Liu, Qun and Wong, Kam-Fai},
  year = {2023},
}