search

NL2Code / CodeR

152 stars 17 forks source link

Does PlanD use ground truth tests? #4

Closed zkx06111 closed 5 months ago

zkx06111 commented 5 months ago

Hi, does the plan D mentioned in your paper use "fail-to-pass" tests that are actually used to evaluate the patches?

If so, this would be kind of unfair because most of the other methods do not use those.

Could you maybe specify how many instances in the paper are solved by Plan D?

NL2Code commented 5 months ago

Hi, please refer to the Issue #2. We will update our arXiv to make this point clear. More details such as logs, results, and trajs can be found in this repo.

Thank your for your questions.