Codegen

[SonWY2]

fig1

[SonWY2]

table 5

[SonWY2]

table 6

[SonWY2]

table 1

[SonWY2]

table 2

[SonWY2]

table 3

[SonWY2]

bench mark data Multi-Turn Programming Benchmark

Format

Each line is a problem expressed in JSON format, consisting of the following fields:

• id: Problem ID
• name: Problem name (cf. Appendix D)
• description: A short description of the problem
• category: Manually labeled problem category
• prompts: A list of template-enabled strings, specifying each step.
• inputs: A list consisting of 5 test case inputs. Each test case is a key-value table mapping the variables (used in the templated prompt) to actual values.
• outputs: A list consisting of 5 test case outputs. Each test case is an expected output value of the program.
• max_gen_length: Maximum number of tokens we set for each turn for the problem. The value is mostly 128 because each turn doesn't require substantial lines of code, but we adjusted a higher number when long generation is expected.

{"prompts":  ["Assign the string \"{A}\" to a variable named \"my_string\".", 
       "Lowercase the given string \"my_string\".", 
       "Assign the distinct characters of the string to a variable named \"chars\".", 
       "Sort these characters in alphabetical order.", 
       "Print the resulting list of characters."  ], 
"inputs": [ {"A": "abcde"}, 
        {"A": "abcdecadeCADE"}, 
        {"A": "aaaaAAAAaaaa"}, 
        {"A": "Jerry jERRY JeRRRY"}, 
        {"A": "ddddc"}  ], 
"outputs": [
        ["a", "b", "c", "d", "e"], 
        ["a", "b", "c", "d", "e"], 
        ["a"], 
        [" ", "e", "j", "r", "y"], 
        ["c", "d"]  ], 
"max_gen_length": 128, 
"category": "string", 
"name": "Sandwich string", 
"description": "Append a string in the middle of another string.", 
"id": "1"}

[SonWY2]

table 4

[SonWY2]

fig 2

[SonWY2]

fig 3

[SonWY2]

fig 4

[SonWY2]

fig 5 (part)

[SonWY2]

fig 6(edited)

[SonWY2]

table 11

[SonWY2]

fig 3 (edited)