search

huangd1999 / EffiBench

[NeurIPS 2024] EffiBench: Benchmarking the Efficiency of Automatically Generated Code
57 stars 12 forks source link

Number of problems #3

Open Antimo05 opened 4 days ago

Antimo05 commented 4 days ago

Hello, first of all I would like to congratulate you for your work, very useful and interesting. I write to you to ask why in the paper it is written that the dataset contains 1000 problems but then in reality in the json file there are 957. Thanks

huangd1999 commented 3 days ago

Hi, I am not sure but when I check the number of problems in the json file, it contains 1000 tasks. Can you provide more details of your results? How did you calculate the number of problems?

image
Antimo05 commented 3 days ago

Yes you're right, I have excluded problems that generate errors for human solutions (as indicated in the previous comment)

huangd1999 commented 3 days ago

Oh, get it. I will address those problems in recent times.