search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
744 stars 193 forks source link

added FIM tokens for bigcode/large-model #66

Closed benlipkin closed 1 year ago

benlipkin commented 1 year ago

bigcode/large-model uses slightly different FIM tokens than bigcode/santacoder. I've integrated these into the eval harness and am currently running the DS1000 benchmark in insertion mode which appears to be working based on partial results so far.