search

open-compass / opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
https://opencompass.org.cn/
Apache License 2.0
3.74k stars 401 forks source link

[Bug] query_per_second does not work #1418

Open a154377713 opened 1 month ago

a154377713 commented 1 month ago

Prerequisite

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

{'CUDA available': False, 'GCC': 'n/a', 'MMEngine': '0.10.4', 'MUSA available': False, 'OpenCV': '4.10.0', 'PyTorch': '2.4.0', 'PyTorch compiling details': 'PyTorch built with:\n' ' - C++ Version: 201703\n' ' - MSVC 192930154\n' ' - Intel(R) oneAPI Math Kernel Library Version ' '2023.1-Product Build 20230303 for Intel(R) 64 ' 'architecture applications\n' ' - Intel(R) MKL-DNN v3.4.2 (Git Hash ' '1137e04ec0b5251ca2b4400a4fd3c667ce843d67)\n' ' - OpenMP 2019\n' ' - LAPACK is enabled (usually provided by ' 'MKL)\n' ' - CPU capability usage: AVX2\n' ' - Build settings: BLAS_INFO=mkl, ' 'BUILD_TYPE=Release, CUDA_VERSION=12.4, ' 'CUDNN_VERSION=9.1.0, ' 'CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, ' 'CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc ' '/Zc:__cplusplus /bigobj /FS /utf-8 ' '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO ' '-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER ' '-DUSE_FBGEMM -DUSE_XNNPACK ' '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 ' '/wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 ' '/wd4804 /wd4273, LAPACK_INFO=mkl, ' 'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, ' 'PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, ' 'USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=OFF, ' 'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, ' 'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, ' 'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, ' 'USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, ' 'USE_ROCM_KERNEL_ASSERT=OFF, \n', 'Python': '3.10.14 | packaged by Anaconda, Inc. | (main, May 6 2024, ' '19:44:50) [MSC v.1916 64 bit (AMD64)]', 'TorchVision': '0.19.0', 'numpy_random_seed': 2147483648, 'opencompass': '0.2.6+', 'sys.platform': 'win32'}

Reproduces the problem - code/configuration sample

from opencompass.models import VLLMwithChatTemplate from mmengine.config import read_base from opencompass.models import Gemini from opencompass.partitioners import NaivePartitioner from opencompass.runners.local_api import LocalAPIRunner from opencompass.tasks import OpenICLInferTask with read_base():

from .lvevalcmrc_mixup.lveval_cmrc_mixup_gen import (

#     LVEval_cmrc_mixup_datasets,
# )
# from .lvevaldureader_mixup.lveval_dureader_mixup_gen import (
#     LVEval_dureader_mixup_datasets,
# )
# from .lvevalfactrecall_en.lveval_factrecall_en_gen import (
#     LVEval_factrecall_en_datasets,
# )
# from .lvevalfactrecall_zh.lveval_factrecall_zh_gen import (
#     LVEval_factrecall_zh_datasets,
# )
# from .lvevalhotpotwikiqa_mixup.lveval_hotpotwikiqa_mixup_gen import (
#     LVEval_hotpotwikiqa_mixup_datasets,
# )
# from .lvevallic_mixup.lveval_lic_mixup_gen import LVEval_lic_mixup_datasets
# from .lvevalloogle_CR_mixup.lveval_loogle_CR_mixup_gen import (
#     LVEval_loogle_CR_mixup_datasets,
# )
# from .lvevalloogle_MIR_mixup.lveval_loogle_MIR_mixup_gen import (
#     LVEval_loogle_MIR_mixup_datasets,
# )
# from .lvevalloogle_SD_mixup.lveval_loogle_SD_mixup_gen import (
#     LVEval_loogle_SD_mixup_datasets,
# )
# from .lvevalmultifieldqa_en_mixup.lveval_multifieldqa_en_mixup_gen import (
#     LVEval_multifieldqa_en_mixup_datasets,
# )
from .datasets.lveval.lvevalmultifieldqa_zh_mixup.lveval_multifieldqa_zh_mixup_gen import (
    LVEval_multifieldqa_zh_mixup_datasets,
)

datasets = sum( (v for k, v in locals().items() if k.endswith('_datasets')), [] )

api_meta_template = dict(round=[ dict(role='HUMAN', api_role='HUMAN'), dict(role='BOT', api_role='BOT', generate=True), ], )

models = [ dict( abbr='gemini-1.5-flash', type=Gemini, path='gemini-1.5-flash', key= '', # The key will be obtained from $GEMINI_API_KEY, but you can write down your key here as well meta_template=api_meta_template, query_per_second=0.1,

max_out_len=100,

    # max_seq_len=2048,
    batch_size=1,
    temperature=1,
)

]

infer = dict( partitioner=dict(type=NaivePartitioner), runner=dict( type=LocalAPIRunner, max_num_workers=1, concurrent_users=1, task=dict(type=OpenICLInferTask)), )

Reproduces the problem - command or script

configs/eval_gemini_1_5_lveval.py --debug

Reproduces the problem - error message

signal.SIGALRM is not available on this platform signal.SIGALRM is not available on this platform 08/13 11:17:55 - OpenCompass - INFO - Current exp folder: outputs\default\20240813_111755 08/13 11:17:55 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored. 08/13 11:17:55 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners 08/13 11:17:55 - OpenCompass - DEBUG - Get class NaivePartitioner from "partitioner" registry in "opencompass" 08/13 11:17:55 - OpenCompass - DEBUG - An NaivePartitioner instance is built from registry, and its implementation can be found in opencompass.partitioners.naive 08/13 11:17:55 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored. 08/13 11:17:55 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored. 08/13 11:17:55 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored. 08/13 11:17:55 - OpenCompass - DEBUG - Additional config: {} 08/13 11:17:55 - OpenCompass - INFO - Partitioned into 1 tasks. 08/13 11:17:55 - OpenCompass - DEBUG - Task 0: [gemini-1.5-flash/LVEval_multifieldqa_zh_mixup_32k] 08/13 11:17:55 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners 08/13 11:17:55 - OpenCompass - DEBUG - Get class LocalAPIRunner from "runner" registry in "opencompass" 08/13 11:17:55 - OpenCompass - DEBUG - An LocalAPIRunner instance is built from registry, and its implementation can be found in opencompass.runners.local_api 08/13 11:17:55 - OpenCompass - DEBUG - Modules of opencompass's task registry have been automatically imported from opencompass.tasks 08/13 11:17:55 - OpenCompass - DEBUG - Get class OpenICLInferTask from "task" registry in "opencompass" 08/13 11:17:55 - OpenCompass - DEBUG - An OpenICLInferTask instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer 08/13 11:17:59 - OpenCompass - INFO - Task [gemini-1.5-flash/LVEval_multifieldqa_zh_mixup_32k] signal.SIGALRM is not available on this platform signal.SIGALRM is not available on this platform C:\Users\PC\anaconda3\envs\opencompass\lib\site-packages\datasets\load.py:1486: FutureWarning: The repository for Infinigence/LVEval contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/Infinigence/LVEval You can avoid this message in future by passing the argument trust_remote_code=True. Passing trust_remote_code=True will be mandatory to load this dataset from the next major release of datasets. warnings.warn( 08/13 11:18:17 - OpenCompass - INFO - Start inferencing [gemini-1.5-flash/LVEval_multifieldqa_zh_mixup_32k] [2024-08-13 11:18:17,031] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process... 2%|▏ | 2/133 [00:15<17:44, 8.12s/it]08/13 11:18:38 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 197 - API key expired. Please renew the API key. 08/13 11:18:38 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 200 - {'error': {'code': 400, 'message': 'API key expired. Please renew the API key.', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'API_KEY_INVALID', 'domain': 'googleapis.com', 'metadata': {'service': 'generativelanguage.googleapis.com'}}]}} 5%|▌ | 7/133 [01:16<22:28, 10.70s/it]08/13 11:19:40 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 197 - An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting 08/13 11:19:40 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 200 - {'error': {'code': 500, 'message': 'An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting', 'status': 'INTERNAL'}} 08/13 11:19:50 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 197 - An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting 08/13 11:19:50 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 200 - {'error': {'code': 500, 'message': 'An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting', 'status': 'INTERNAL'}} 5%|▌ | 7/133 [01:34<28:18, 13.48s/it]

Other information

I changed the query_per_second value but it doesn't work

tonysy commented 1 month ago

To improve the speed, you can increase the batch_size.