bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
709 stars 183 forks source link

Update stop tokens for mbpp and humaneval #125

Closed VikParuchuri closed 10 months ago

VikParuchuri commented 11 months ago

When training a model on markdown/notebook data, the model can output backticks after a code block. This will cause the code to not run, even though the solution might be valid.

This updates the mpbb and humaneval stop tokens to account for this behavior.