Is there a way to evaluate performance between alpaca and alpaca-lora?

[Jeffwan]

I am just curious whether there's a scientific way to compare the performance between alpaca and alpaca -lora? Does the community have some evaluation scripts to run?

[claysauruswrecks]

@tloen - We can probably use this issue to track adding an eval harness to the pipeline: https://github.com/EleutherAI/lm-evaluation-harness

[claysauruswrecks]

https://github.com/openai/evals

[claysauruswrecks]

https://github.com/bigcode-project/bigcode-evaluation-harness

[hujunchao]

I am also instrested in this issue.

[JACKHAHA363]

Hi is there update?

[gururise]

Hi is there update?

Added initial squad benchmarks, working on piqa.

[ndvbd]

Also interested