Question about the total number of methods in the benchmark

FudanSELab / ClassEval

Benchmark ClassEval for class-level code generation.

MIT License

108 stars 5 forks source link

Question about the total number of methods in the benchmark #6

Closed ztwater closed 1 month ago

ztwater commented 3 months ago

Hi, I noticed that the paper declares that the ClassEval benchmark includes 412 methods. However, I only got 410 of them in your latest data (By simply collecting all methods in the methods_info list). Could you please explain which two methods are excluded and why? Many thanks!

wkx228 commented 2 months ago

We have updated our benchmark, so the statistics of the latest version may differ slightly from those given in the paper.

DXY-lemon commented 2 months ago

We have updated the README with the latest statistics, and thank you for carefully pointing out the issue! We removed two methods related to file handling due to testability issue.

ztwater commented 1 month ago

Thanks for your responses!