EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.61k stars 1.75k forks source link

Add TensorRT-LLM support #1910

Open taewan2002 opened 4 months ago

taewan2002 commented 4 months ago

I am trying to run the benchmarking on an Nvidia Orin 64GB machine due to lack of GPU resources, but it is too slow, so I would appreciate it if you could apply TensorRT-LLM. 🤣

haileyschoelkopf commented 4 months ago

Hello!

We don't currently support TRT-LLM, though we do support VLLM which should improve over HF performance.

We'd however welcome a contribution adding TRT-LLM!