exact commandline to reproduce leaderboard

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.

Apache License 2.0

781 stars 208 forks source link

exact commandline to reproduce leaderboard #162

Open Derekglk opened 10 months ago

Derekglk commented 10 months ago

Wonderful Bigcode collaborators,

Can we add a new column to the Big Code Leaderboard which describes the exact command line to reproduce the scores? I saw many similar questions asking about how to reproduce the scores, and the differences against the paper. This new column will definitely help.

Thanks!

loubnabnl commented 10 months ago

Good suggestion, will add it to the todo list

Anindyadeep commented 9 months ago

Hi, I just saw this issue, but does the only variable is the model_name here to reproduce results? And if that is the case, do we require a whole column?