bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
744 stars 193 forks source link

[WIP] Add the BugRepair Task evaluation #56

Closed keyboardAnt closed 1 year ago

keyboardAnt commented 1 year ago

[Work in Progress] Add the BugRepair Task evaluation over a dataset of ~2500 repairs in Python (from https://arxiv.org/pdf/2105.12787.pdf).