bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
744 stars 193 forks source link

requirements.txt doesn't support newer models (KeyError) #77

Closed sedrickkeh closed 1 year ago

sedrickkeh commented 1 year ago

(Related issue here: #73) The requirements.txt file lists transformers==4.25.1, which doesn't support a lot of the newer models such as bigcode/starcoder and huggyllama/llama-7b (it gives errors such as KeyError: 'gpt_bigcode'). This should be simple to fix from the user side (just install a newer version of transformers), but just thought I'd flag it here since it's probably best if the requirements.txt can accommodate these newer models.

loubnabnl commented 1 year ago

Thank you for flagging this! should be fixed with this PR