Closed zhimin-z closed 7 months ago
This is a perpetual project, the latest reports can be seen on the leaderboard page: https://huggingface.co/spaces/mike-ravkine/can-ai-code-results
No plans for a paper I'm afraid, I am a hobbyist with no academic affiliation.
So the leaderboard is basically the de facto report?
@zhimin-z Yes the leaderboard is the report.
Note there two interviews in there, "junior-v2" is the original simple one, which most modern models can easily pass, and "senior" is a new much more difficult one that most models are failing. I am still working on senior, need to add more tasks.
All the evaluation data is available inside the results/ folder of the repo if you'd like to dig deeper into why a particular model has the score it does.
This is a perpetual project, the latest reports can be seen on the leaderboard page: https://huggingface.co/spaces/mike-ravkine/can-ai-code-results
No plans for a paper I'm afraid, I am a hobbyist with no academic affiliation.