Program repair

[keyboardAnt]

• Add an Automatic Program Repair (APR) Task evaluation over a dataset of ~2500 repairs in Python, inspired by https://arxiv.org/abs/2105.12787 and https://carper.ai/diff-models-a-new-way-to-edit-code/.
• Support generating from SaaS APIs (like OpenAI).

[andre15silva]

Nice to see program repair here! :)

Quick question: Any reason why you consider PyPiBugs only? The dataset contains only 7 types of rather simple bugs.

I would argue that benchmarks like https://github.com/rjust/defects4j (the "gold standard" in academia until now) or https://github.com/giganticode/run_bug_run (recent multi-language executable benchmark) are more interesting and challenging. WDYT?

[keyboardAnt]

Nice to see program repair here! :)

Quick question: Any reason why you consider PyPiBugs only? The dataset contains only 7 types of rather simple bugs.

I would argue that benchmarks like https://github.com/rjust/defects4j (the "gold standard" in academia until now) or https://github.com/giganticode/run_bug_run (recent multi-language executable benchmark) are more interesting and challenging. WDYT?

@andre15silva, thanks for the question. The suggested task could be used with any similar datasets.

[keyboardAnt]

Thanks for these contributions Nadav! I left some high level comments, when addressed we can go through the benchmark details. Could you also explain how this benchmark works? you said it's an inspiration from Carper's and MSFT's benchmarks, does it follow the same implementation as carper? If not what's different and how did you build the dataset since we usually port benchmarks from existing repositories/papers.

Thank you for your queries, @loubnabnl. This task indeed mirrors the benchmark from Carper's and MSFT's. It has been seamlessly integrated into our existing codebase and utilizes a pre-existing dataset.