search

ShishirPatil / gorilla

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
https://gorilla.cs.berkeley.edu/
Apache License 2.0
11.52k stars 1.01k forks source link

[BFCL] Non_Live Overall Accuracy、Non-Live AST Accuracy、AST Summary、Non-Live Exec Accuracy、Live Accurac这些指标数据如何评估某个模型的优劣区别？ #788

Closed belief888 closed 3 days ago

belief888 commented 3 days ago

bfcl evaluate --model Qwen/Qwen2.5-7B-Instruct --test-category simple 执行这条命令有输出结果，其中指标数据如下： Rank Model Live Overall Acc AST Summary Python Simple AST Python Multiple AST Python Parallel AST Python Parallel Multiple AST Irrelevance Detection Relevance Detection 1 Qwen2.5-7B-Instruct (Prompt) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%

Rank Model Multi Turn Overall Acc Base Miss Func Miss Param Long Context 1 Qwen2.5-7B-Instruct (Prompt) 0.00% 0.00% 0.00% 0.00% 0.00%

Rank Model Non_Live Overall Acc AST Summary Exec Summary Simple AST Python Simple AST Java Simple AST JavaScript Simple AST Multiple AST Parallel AST Parallel Multiple AST Simple Exec Python Simple Exec REST Simple Exec Multiple Exec Parallel Exec Parallel Multiple Exec Irrelevance Detection 1 Qwen2.5-7B-Instruct (Prompt) 3.55% 7.98% 0.00% 31.92% 95.75% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%

Rank Overall Acc Model Model Link Cost ($ Per 1k Function Calls) Latency Mean (s) Latency Standard Deviation (s) Latency 95th Percentile (s) Non-Live AST Acc Non-Live Simple AST Non-Live Multiple AST Non-Live Parallel AST Non-Live Parallel Multiple AST Non-Live Exec Acc Non-Live Simple Exec Non-Live Multiple Exec Non-Live Parallel Exec Non-Live Parallel Multiple Exec Live Acc Live Simple AST Live Multiple AST Live Parallel AST Live Parallel Multiple AST Multi Turn Acc Multi Turn Base Multi Turn Miss Func Multi Turn Miss Param Multi Turn Long Context Relevance Detection Irrelevance Detection Organization License 1 1.18% Qwen2.5-7B-Instruct (Prompt) https://huggingface.co/Qwen/Qwen2.5-7B-Instruct N/A 6.55 3.23 12.51 7.98% 31.92% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Qwen apache-2.0

请问根据以上输出的指标数据，如何统计各个模型的最新score分数？有没有具体的文档分析？

HuanzhiMao commented 3 days ago

如何统计各个模型的最新score分数？

我有点没明白你的问题；模型最新的score就是 data_xxx.csv里的数据

有没有具体的文档分析？

这个你需要看每一项的score文件，看他们的error log去分析在哪些情况会错

belief888 commented 3 days ago

这个分析了。