feat(quality): add aider's code editing bechmarks as a source of quaity scores

stencila / evals

Apache License 2.0

0 stars 0 forks source link

Open nokome opened 4 weeks ago

nokome commented 4 weeks ago

brettc commented 4 weeks ago

Making some notes here.

This outlines the details of the benchmarking: https://github.com/Aider-AI/aider/blob/main/benchmark/README.md
There are results written to the GitHub repo in YAML format: https://github.com/Aider-AI/aider/tree/main/aider/website/_data
The key metrics are the pass_rate_# fields. There may be multiple, depending on the passes.