search

FlagOpen / TACO

Apache License 2.0
146 stars 8 forks source link

How to construct the appropriate output? #2

Closed QuaY316 closed 10 months ago

QuaY316 commented 11 months ago

I try to evaluate codellama-7b using the easy difficulty of the TACO dataset. But I find that the output code needs to meet certain specifications in order to pass the test cases, like s = input()\nprint(s.swapcase()) not def solve(s):\n return s.swapcase() How to construct the appropriate output?

bowen92 commented 10 months ago

The difference is the format to use. Actually we should claim the format to use when given the question.

s = input()\nprint(s.swapcase()) is the "Standard Input format" while the code def solve(s):\n return s.swapcase() is "Call-Based format"

We double-checked our code and update the prompt in generation.py, please use the updated version. Let us know if you have other problem.