Closed IQ179 closed 1 year ago
https://github.com/bigcode-project/bigcode-evaluation-harness/blob/5b7e7235d97a45ce7881d8be478c58609dfe93fb/lm_eval/tasks/multiple.py#L39
If "bs" in the Language list means bash, I think it should be changed to "sh". Even if you look at the dataset name in hf, it is sh, so the current code does not load the dataset properly when working with All_task.
https://github.com/bigcode-project/bigcode-evaluation-harness/blob/5b7e7235d97a45ce7881d8be478c58609dfe93fb/lm_eval/tasks/multiple.py#L39
If "bs" in the Language list means bash, I think it should be changed to "sh". Even if you look at the dataset name in hf, it is sh, so the current code does not load the dataset properly when working with All_task.