search

TianjinYellow / EdgeDeviceLLMCompetition-Starting-Kit

23 stars 4 forks source link

dataset name #3

Open wangziyannb opened 1 week ago

wangziyannb commented 1 week ago

In the readme file, the eval dataset contains commonseqa_gen. image

I got this error when trying to set up starting kit:

ValueError: The provided pattern matches 0 or more than one config. Please verify your pattern and try again. You may use tools/list_configs.py to list or locate the configurations. +------------------------+ Not matched patterns
commonseqa_gen

+------------------------+

Is this a typo? The supported configs related to commensenseqa are:

| commonsenseqa_gen | configs/datasets/commonsenseqa/commonsenseqa_gen.py | | commonsenseqa_gen_1da2d0 | configs/datasets/commonsenseqa/commonsenseqa_gen_1da2d0.py | | commonsenseqa_gen_c946f2 | configs/datasets/commonsenseqa/commonsenseqa_gen_c946f2.py | | commonsenseqa_ppl | configs/datasets/commonsenseqa/commonsenseqa_ppl.py | | commonsenseqa_ppl_3e9f2d | configs/datasets/commonsenseqa/commonsenseqa_ppl_3e9f2d.py | | commonsenseqa_ppl_5545e2 | configs/datasets/commonsenseqa/commonsenseqa_ppl_5545e2.py | | commonsenseqa_ppl_716f78 | configs/datasets/commonsenseqa/commonsenseqa_ppl_716f78.py | | commonsenseqa_ppl_c49e77 | configs/datasets/commonsenseqa/commonsenseqa_ppl_c49e77.py | | commonsenseqa_ppl_e51e32 | configs/datasets/commonsenseqa/commonsenseqa_ppl_e51e32.py | | commonsenseqacn_gen | configs/datasets/commonsenseqa_cn/commonsenseqacn_gen.py | | commonsenseqacn_gen_d380d0 | configs/datasets/commonsenseqa_cn/commonsenseqacn_gen_d380d0.py | | commonsenseqacn_ppl | configs/datasets/commonsenseqa_cn/commonsenseqacn_ppl.py | | commonsenseqacn_ppl_971f48 | configs/datasets/commonsenseqa_cn/commonsenseqacn_ppl_971f48.py |

Do you mean commonsenseqa_gen?

TianjinYellow commented 1 week ago

Hi Wangziyannb，

Yes, this is a typo. you are right, it is commonsenseqa_gen. Thanks for pointing it out.

Best,

wangziyannb commented 1 week ago

By the way, we should use pip (or pip3) instead of Pip

wangziyannb commented 1 week ago

There is another problem related to bbh dataset. I checked opencompass repo, and there should be a folder named lib_prompt here.

https://github.com/open-compass/opencompass/tree/main/configs/datasets/bbh image

Which is missed and will raise a file not found issue when evaluating with bbh. image image

This folder can be found in the zip file mentioned in readme somehow. Maybe they fixed some problems in recent commits.

TianjinYellow commented 1 week ago

Thank you for your kind reminder. DId you install opencompass from our repository? It should work well for bbh_gen evaluation.

Best,