search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
698 stars 180 forks source link

Support for HumanEval-Infilling Benchmark #203

Closed Hambaobao closed 3 months ago

Hambaobao commented 4 months ago

Thank you very much for providing the evaluation code. I would like to inquire if there are any plans to include the HumanEval-Infilling Benchmark evaluation in the current tasks.

loubnabnl commented 4 months ago

Not at the moment, we already support a similar benchmark SantaCoder-FIM https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/docs#santacoder-fim