search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
744 stars 193 forks source link

chat bug fixer for humaneval-x-bugs #57

Closed mitya52 closed 1 year ago

mitya52 commented 1 year ago

I tested code only on HumanEval for python, but seems it will work on other languages also.

Results over one iteration of HumanEval: {'pass@1': 0.7682926829268293}

mitya52 commented 1 year ago

Rust fixed + cpp and js parser improvement

Muennighoff commented 1 year ago

I think we're done here - Merging this if okay with you?

mitya52 commented 1 year ago

yes, that's ok!