Match llama-3.1-8B results (using lm-eval-harness) from litgpt/huggingface api

whittle-org / whittle

Python library to compress LitGPT models for resource efficient inference.

https://whittle-org.github.io/whittle/latest/

Apache License 2.0

10 stars 4 forks source link

Match llama-3.1-8B results (using lm-eval-harness) from litgpt/huggingface api #129

Closed rheasukthanker closed 1 month ago

rheasukthanker commented 1 month ago

Describe the bug The performance (for example on arc_easy) from litgpt (using hf api) and using our whittle API does not match exactly for Llama-3.1-8B. This is likely due to issue with RoPE https://github.com/Lightning-AI/litgpt/pull/1745 and needs to be investigated further