search

hendrycks / apps

APPS: Automated Programming Progress Standard (NeurIPS 2021)
MIT License
414 stars 55 forks source link

About Solutiions' validity #26

Closed MrBlack0220 closed 1 year ago

MrBlack0220 commented 1 year ago

Thanks for creating this amazing datasets. I want to ask if any solution in Solutions.json is strictly correct (can pass all unit tests in input_output.json )? Do you check it? If not，how do you ensure the correctness of the solution? Thanks!

xksteven commented 1 year ago

All the solutions were pulled from websites with thousands of users. The answers are all valid as they've been checked by the writers and users. The solutions are valid in so far as they are able to produce the correct answers given the corresponding inputs. We worked to ensure that all the ones we gathered are valid and match that description. Beyond that we also excluded solutions which took to long to compute. For more details feel free to read the corresponding paper as to more specifics about the data collection process.

Hope that answers your question.