search

apartresearch / evaluations-starter

How to get started in evaluations and demonstrations research for dangerous capabilities
MIT License
5 stars 1 forks source link

Update to starter colab notebook for lm-eval-harness #3

Open sepiatone opened 3 months ago

sepiatone commented 3 months ago

There has been a refactor of lm-eval-harness a few months ago. The starter colab notebook for lm-eval-harness, linked to from this section has to be slightly updated to reflect this.

Instead of

!python lm-evaluation-harness/main.py \
    --model gpt2 \
    --tasks truthfulqa_gen \
    --device 0

we have

!lm-eval --model hf --model_args pretrained=EleutherAI/Pythia-70m --tasks truthfulqa_gen --device 0
jas-ho commented 3 months ago

thanks, I'd be glad if you submitted a PR!

sepiatone commented 3 months ago

thanks, I'd be glad if you submitted a PR!

will do that!