search

google-deepmind / code_contests

Apache License 2.0
2.04k stars 200 forks source link

Details of codeforces generated tests #33

Open manleyroberts opened 1 year ago

manleyroberts commented 1 year ago

3.2.1. of the arxiv version of the paper, page 8 mentions that the Codeforces generated tests are produced through a complex procedure based on mutating inputs and verifying agreement of correct solutions. Is there any plan to release the code for this generation procedure? Replication and extension of the codeforces portion of the code contests dataset might require the application of this process to more recent problems.

Thanks!

felixgimeno commented 11 months ago

We don't have plans to release the code for the generated tests. The overall idea is to replace random subset of pre-existing inputs with similar elements (strings with strings, numbers with numbers), then run code solutions that are known to be good, and seeing if there is consensus in the outputs for the same input, if there is consensus then we regard that (input, output) pair as a generated test. You can use any method to generate the inputs. Usually the challenge is filtering the "inputs" that are actually invalid for the problem, and for this we run the correct code solutions, but you can use any method for this for example manual checking.