-
Nice to see the newly added rotation support!
I saw below in the comment.
https://github.com/HandH1998/QQQ/blob/bafdb00fd901cd13f668f4028d7891276ded3bb2/examples/quant_model.py#L70
Can you share …
-
**Describe the bug**
Evaluation only works on safetensors at the moment for MMLU and MMLUBranch, but we are passing a GGUF file by default and there is no error handling for it
**To Reproduce**
S…
-
This is a very valuable project for research. I tried to download it. The demo cpp llama1.1b w4a8 can run at 18 t/s. Although the output of the model is not what I asked, it seems that there is no pro…
-
Models that are open-source and/or used via `local-completions`, as well as Claude, allow one to "prefill" the start of the assistant's response to a given input: https://docs.anthropic.com/en/docs/bu…
-
Dear vlievin,
I am writing to request access to the MMLU dataset, which I understand is available through GitHub.
As a researcher in the field of machine learning, I am interested in using the M…
-
When I run the examples/evaluator. py file, the code runs from awq.evaluation import(
evaluate_perplexity,
eval_librispeech,
eval_mmlu,
eval_humaneval,
eval_kl_divergence,)An ImportError occurred…
-
### System Info
GPUs: A100, 4 GPUs (40 GB memory)
Release: tensorrt-llm 0.9.0
### Who can help?
@Tracin
### Information
- [X] The official example scripts
- [ ] My own modified scrip…
vnkc1 updated
2 months ago
-
### System Info
- `transformers` version: 4.44.1
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4…
-
### 描述该功能
求配置mmlu_pro数据集的代码逻辑~
### 是否希望自己实现该功能?
- [ ] 我希望自己来实现这一功能,并向 OpenCompass 贡献代码!
-
**Is your feature request related to a problem? Please describe.**
I'm trying to evaluate a local LLM model using Exllamav2 and Deepbench's support for the [MMLU dataset](https://docs.confident-ai.co…