search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
710 stars 183 forks source link

Support `Salesforce/codet5p-220m` and other `T5ForConditionalGeneration` models #102

Closed keyboardAnt closed 3 weeks ago

keyboardAnt commented 1 year ago

AutoModelForCausalLM.from_pretrained(Salesforce/codet5p-220m) raises ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM.. Consider utilizing AutoModelForSeq2SeqLM. Here is a quick draft example.

loubnabnl commented 8 months ago

This was added but only tested for instructcodet5p and HumanEvalPack here https://github.com/bigcode-project/bigcode-evaluation-harness/blob/9d17482e38eaeddee5e27488229a893dc59d42e1/main.py#L274 If more tests are done for other tasks we can generalize it