search

evalplus / repoqa

RepoQA: Evaluating Long-Context Code Understanding
https://evalplus.github.io/repoqa.html
Apache License 2.0
96 stars 3 forks source link

feat: comment analysis #19

Closed JialeTomTian closed 5 months ago

JialeTomTian commented 5 months ago

Support for comment analysis for the 5 languages. Tested it on three languages (rust, python and java). Performed a couple of quick experiments, here is the visualization

Screen Shot 2024-04-11 at 3 45 07 PM Screen Shot 2024-04-11 at 3 45 45 PM Screen Shot 2024-04-11 at 3 47 15 PM Screen Shot 2024-04-11 at 4 14 25 PM Screen Shot 2024-04-11 at 4 51 45 PM

This PR aims to close #15

ganler commented 5 months ago

This is great. Overall I feel most functions I see have a small comment ratio and in this sense, I might set a threshold of 0.2.

I think let's not worry about the comments above the function for now. It is a good point but I feel it is a bit of a small ROI. So let's just use this for the short term. :) Thanks again for the great work!