search

mlabonne / llm-autoeval

Automatically evaluate your LLMs in Google Colab
MIT License
460 stars 77 forks source link

How to run it locally? #16

Open JamesKnight0001 opened 4 months ago

JamesKnight0001 commented 4 months ago

How Do I run it locally? And can it run .gguf files?

mlabonne commented 4 months ago

@CultriX-Github did a really cool fork that can run locally: https://github.com/CultriX-Github/llm-autoeval

It doesn't support GGUF versions, however. Might be added in the future.

CultriX-Github commented 4 months ago

To elaborate on this, you would need to:

Steps:

  1. Fork the repo: https://github.com/CultriX-Github/llm-autoeval.git and run t he runpod.sh script which will ask you for some of the variables that are usually being set by the Google Colab script that starts the Runpod instance.

git clone https://github.com/CultriX-Github/llm-autoeval && cd llm-autoeval && bash runpod.sh

  1. Wait for the runpod.sh script to complete.
  2. Profit

Optionally, you could start the script in a screen sessions so you don't lose your benchmarking progress if you get disconnected. To do so, simply run:

git clone https://github.com/CultriX-Github/llm-autoeval && cd llm-autoeval && apt update -y && apt install -y screen && screen -S autoeval -m bash runpod.sh