How to eval GEAR with lm-eval framework?

opengear-project / GEAR

GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM

MIT License

128 stars 10 forks source link

How to eval GEAR with lm-eval framework? #6

Closed ThisisBillhe closed 1 month ago

ThisisBillhe commented 4 months ago

Hi Hao Kang, sorry to interrupt. I see there is a lm-harness dir, however I can not find a script or guide on how to evaluate GEAR with lm_eval. Am I missing something?

HaoKang-Timmy commented 3 months ago

Please look at this. I am currently updating kernel support for GEAR so the whole guidance will be refined in future. https://github.com/opengear-project/GEAR/blob/main/lm-harness/shfile_example.sh

HaoKang-Timmy commented 1 month ago

Done