-
Is there support for multi-node MMLU evaluation ?
I'd be ok with trying experimental stuff, etc.
-
Hi thanks for providing such wonderful evaluation toolkit.
I was wondering why evaluation on `mmlu_generative` returns 0 accuracy whenever what models I try (pythia, qwen).
I understand it as …
-
MMLU-Pro info is as below:
1.paper: https://arxiv.org/pdf/2406.01574
2.huggingface: https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro
3.GitHub: https://github.com/TIGER-AI-Lab/MMLU-Pro
-
I'm having an issue where the content of the results is coming out empty. Here is a snippet of the code and result:
The result for task_queue[99] is:
``` python
Info(name='task', author='User',…
-
The bug I met is similar to #203. I'm trying to reproduce the evaluation results of [ablation model trained on FineWeb](https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-v1/tree/main), using…
-
Hi, I noticed that in some templates of MMLU, it seems that the part of the prompt "The following are multiple choice questions (with answers) about {topic}." is being placed in "input_format", when i…
-
Steps to recreate:
Launch mmlu on an instance with multiple gpus. Run:
ilab model evaluate --model models/instructlab/granite-7b-lab --benchmark mmlu
Only 1 gpu is consumed. Adjusting batch…
-
Hi,
I am trying to run some LLMs (currently trying openai models) on MMLU. My first question is which configuration is the standard setup (5 shot without CoT)? What does flan mean in some of the c…
-
Hi,
I fine-tuned the LLaMA-3-8b-Instruct on "llama3-ultrafeedback-armorm" in different gradient accumulation (other hyperparameters are the same with llama-3-8b-instruct-simpo-v2.yaml). For fine-t…
-
### Describe the feature
Huggingface : https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro?row=0
### Will you implement it?
- [ ] I would like to implement this feature and create a PR!