search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
744 stars 193 forks source link

Support `transformers.pipeline(model=...)` models like `HuggingFaceH4/starchat-beta` #89

Closed keyboardAnt closed 1 year ago

keyboardAnt commented 1 year ago

I couldn't load HuggingFaceH4/starchat-beta using our existing codebase on a single machine with multiple 40GB GPUs. See loading instructions on the model card.

arjunguha commented 1 year ago

I believe this can be fixed by running pip install --upgrade transformers

loubnabnl commented 1 year ago

Yes like arjun said this isn't an evaluation harness issue you need to upgrade your transformers version to >=4.28.1 where the gpt_bigcode architecture was introduced to be able to load the model.