search

mlabonne / llm-autoeval

Automatically evaluate your LLMs in Google Colab
MIT License
527 stars 82 forks source link

log running time? #1

Closed gblazex closed 6 months ago

gblazex commented 8 months ago

I love how simply this runs.

it would be useful to log how long the test took to run.

Maybe an optional checkbox on the colab and the code handles it?

gblazex commented 8 months ago

also CodeNinja and xDan 7B models fail to run. There is no error logged to gist or anywhere that I can see. (gist just logs "File not found" as results of tests)

mlabonne commented 8 months ago

Thanks! Yeah I thought about it too, I'll add the runtime very soon.

You can run the debug mode to check the logs. In most cases, the model doesn't have a tokenizer in its repo.

mlabonne commented 8 months ago

It's added: https://github.com/mlabonne/llm-autoeval/pull/2