search

modelscope / eval-scope

A streamlined and customizable framework for efficient large model evaluation and performance benchmarking
Apache License 2.0
110 stars 14 forks source link
evaluation llm performance

readme

English | 简体中文

Introduction

Large Language Model (LLMs) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the Eval-Scope framework, which includes the following components and features:

Features

News

Installation

Install with pip

  1. create conda environment

    conda create -n eval-scope python=3.10
conda activate eval-scope

  2. Install Eval-Scope

    pip install llmuses

Install from source code

  1. Download source code

    git clone https://github.com/modelscope/eval-scope.git

  2. Install dependencies

    cd eval-scope/
pip install -e .

Quick Start

Simple Evaluation

command line with pip installation:

python -m llmuses.run --model ZhipuAI/chatglm3-6b --template-type chatglm3 --datasets arc --limit 100

command line with source code:

python llmuses/run.py --model ZhipuAI/chatglm3-6b --template-type chatglm3 --datasets mmlu ceval --limit 10

Parameters:

Evaluation with Model Arguments

python llmuses/run.py --model ZhipuAI/chatglm3-6b --template-type chatglm3 --model-args revision=v1.0.2,precision=torch.float16,device_map=auto --datasets mmlu ceval --use-cache true --limit 10
python llmuses/run.py --model qwen/Qwen-1_8B --generation-config do_sample=false,temperature=0.0 --datasets ceval --dataset-args '{"ceval": {"few_shot_num": 0, "few_shot_random": false}}' --limit 10

Parameters:

Note: you can use following command to check the template type list of the model:

from llmuses.models.template import TemplateType
print(TemplateType.get_template_name_list())

Evaluation Backend

Eval-Scope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:

1. OpenCompass Eval-Backend

To facilitate the OpenCompass as an evaluation backend, we have customized the OpenCompass codebase and named it ms-opencompass. This version enhances the configuration and execution of evaluation tasks based on the original version and supports installation via PyPI, allowing users to initiate lightweight OpenCompass evaluation tasks through Eval-Scope.

Installation
# Install eval-scope
pip install llmuses>=0.4.0

# Install ms-opencompass
pip install ms-opencompass

Data Preparation

Available datasets from OpenCompass backend:

'obqa', 'AX_b', 'siqa', 'nq', 'mbpp', 'winogrande', 'mmlu', 'BoolQ', 'cluewsc', 'ocnli', 'lambada', 'CMRC', 'ceval', 'csl', 'cmnli', 'bbh', 'ReCoRD', 'math', 'humaneval', 'eprstmt', 'WSC', 'storycloze', 'MultiRC', 'RTE', 'chid', 'gsm8k', 'AX_g', 'bustm', 'afqmc', 'piqa', 'lcsts', 'strategyqa', 'Xsum', 'agieval', 'ocnli_fc', 'C3', 'tnews', 'race', 'triviaqa', 'CB', 'WiC', 'hellaswag', 'summedits', 'GaokaoBench', 'ARC_e', 'COPA', 'ARC_c', 'DRCD'

Refer to OpenCompass datasets

You can use the following code to list all available datasets:

from llmuses.backend.opencompass import OpenCompassBackendManager
print(f'** All datasets from OpenCompass backend: {OpenCompassBackendManager.list_datasets()}')

Dataset download:

Unzip the file and set the path to the data directory in current work directory.

Model serving

We use ModelScope swift to deploy model services, see: ModelScope swift

# Install ms-swift
pip install ms-swift

# Deploy model
CUDA_VISIBLE_DEVICES=0 swift deploy --model_type llama3-8b-instruct --port 8000

Model evaluation

Refer to example: example_eval_swift_openai_api to configure and execute the evaluation task:

python examples/example_eval_swift_openai_api.py

Local Dataset

You can use local dataset to evaluate the model without internet connection.

1. Download and unzip the dataset

# set path to /path/to/workdir
wget https://modelscope.oss-cn-beijing.aliyuncs.com/open_data/benchmark/data.zip
unzip data.zip

2. Use local dataset to evaluate the model

python llmuses/run.py --model ZhipuAI/chatglm3-6b --template-type chatglm3 --datasets arc --dataset-hub Local --dataset-args '{"arc": {"local_path": "/path/to/workdir/data/arc"}}'  --limit 10

# Parameters:
# --dataset-hub: dataset sources: `ModelScope`, `Local`, `HuggingFace` (TO-DO)  default to `ModelScope`
# --dataset-args: json format, key is the dataset name, value should be args for the dataset

3. (Optional) Use local mode to submit evaluation task

# 1. Prepare the model local folder, the folder structure refers to chatglm3-6b, link: https://modelscope.cn/models/ZhipuAI/chatglm3-6b/files
# For example, download the model folder to the local path /path/to/ZhipuAI/chatglm3-6b

# 2. Execute the offline evaluation task
python llmuses/run.py --model /path/to/ZhipuAI/chatglm3-6b --template-type chatglm3 --datasets arc --dataset-hub Local --dataset-args '{"arc": {"local_path": "/path/to/workdir/data/arc"}}' --limit 10

Use run_task function

1. Configuration

import torch
from llmuses.constants import DEFAULT_ROOT_CACHE_DIR

# Example configuration
your_task_cfg = {
        'model_args': {'revision': None, 'precision': torch.float16, 'device_map': 'auto'},
        'generation_config': {'do_sample': False, 'repetition_penalty': 1.0, 'max_new_tokens': 512},
        'dataset_args': {},
        'dry_run': False,
        'model': 'ZhipuAI/chatglm3-6b',
        'template_type': 'chatglm3', 
        'datasets': ['arc', 'hellaswag'],
        'work_dir': DEFAULT_ROOT_CACHE_DIR,
        'outputs': DEFAULT_ROOT_CACHE_DIR,
        'mem_cache': False,
        'dataset_hub': 'ModelScope',
        'dataset_dir': DEFAULT_ROOT_CACHE_DIR,
        'stage': 'all',
        'limit': 10,
        'debug': False
    }

2. Execute the task

from llmuses.run import run_task

run_task(task_cfg=your_task_cfg)

Arena Mode

The Arena mode allows multiple candidate models to be evaluated through pairwise battles, and can choose to use the AI Enhanced Auto-Reviewer (AAR) automatic evaluation process or manual evaluation to obtain the evaluation report. The process is as follows:

1. Env preparation

a. Data preparation, the question data format refers to: llmuses/registry/data/question.jsonl
b. If you need to use the automatic evaluation process (AAR), you need to configure the relevant environment variables. Taking the GPT-4 based auto-reviewer process as an example, you need to configure the following environment variables:
    > export OPENAI_API_KEY=YOUR_OPENAI_API_KEY

2. Configuration files

Refer to : llmuses/registry/config/cfg_arena.yaml
Parameters:
    questions_file: question data path
    answers_gen: candidate model prediction result generation, supports multiple models, can control whether to enable the model through the enable parameter
    reviews_gen: evaluation result generation, currently defaults to using GPT-4 as the Auto-reviewer, can control whether to enable this step through the enable parameter
    elo_rating: ELO rating algorithm, can control whether to enable this step through the enable parameter, note that this step depends on the review_file must exist

3. Execute the script

#Usage:
cd llmuses

# dry-run mode
python llmuses/run_arena.py -c registry/config/cfg_arena.yaml --dry-run

# Execute the script
python llmuses/run_arena.py --c registry/config/cfg_arena.yaml

4. Visualization

# Usage:
streamlit run viz.py -- --review-file llmuses/registry/data/qa_browser/battle.jsonl --category-file llmuses/registry/data/qa_browser/category_mapping.yaml

Single Model Evaluation Mode

In this mode, we only score the output of a single model, without pairwise comparison.

1. Configuration file

Refer to: llmuses/registry/config/cfg_single.yaml
Parameters:
    questions_file: question data path
    answers_gen: candidate model prediction result generation, supports multiple models, can control whether to enable the model through the enable parameter
    reviews_gen: evaluation result generation, currently defaults to using GPT-4 as the Auto-reviewer, can control whether to enable this step through the enable parameter
    rating_gen: rating algorithm, can control whether to enable this step through the enable parameter, note that this step depends on the review_file must exist

2. Execute the script

#Example:
python llmuses/run_arena.py --c registry/config/cfg_single.yaml

Baseline Model Comparison Mode

In this mode, we select the baseline model, and compare other models with the baseline model for scoring. This mode can easily add new models to the Leaderboard (just need to run the scoring with the new model and the baseline model).

1. Configuration file

Refer to: llmuses/registry/config/cfg_pairwise_baseline.yaml
Parameters:
    questions_file: question data path
    answers_gen: candidate model prediction result generation, supports multiple models, can control whether to enable the model through the enable parameter
    reviews_gen: evaluation result generation, currently defaults to using GPT-4 as the Auto-reviewer, can control whether to enable this step through the enable parameter
    rating_gen: rating algorithm, can control whether to enable this step through the enable parameter, note that this step depends on the review_file must exist

2. Execute the script

# Example:
python llmuses/run_arena.py --c registry/config/cfg_pairwise_baseline.yaml

Datasets list

DatasetName Link Status Note
mmlu mmlu Active
ceval ceval Active
gsm8k gsm8k Active
arc arc Active
hellaswag hellaswag Active
truthful_qa truthful_qa Active
competition_math competition_math Active
humaneval humaneval Active
bbh bbh Active
race race Active
trivia_qa trivia_qa To be intergrated

Leaderboard

The LLM Leaderboard aims to provide an objective and comprehensive evaluation standard and platform to help researchers and developers understand and compare the performance of models on various tasks on ModelScope.

Leaderboard

Experiments and Results

Experiments

Model Serving Performance Evaluation

Perf

TO-DO List