Closed SonWY2 closed 1 year ago
table 5
table 6
table 1
table 2
table 3
Each line is a problem expressed in JSON format, consisting of the following fields:
id
: Problem IDname
: Problem name (cf. Appendix D)description
: A short description of the problemcategory
: Manually labeled problem categoryprompts
: A list of template-enabled strings, specifying each step.inputs
: A list consisting of 5 test case inputs. Each test case is a key-value table mapping the variables (used in the templated prompt) to actual values.outputs
: A list consisting of 5 test case outputs. Each test case is an expected output value of the program.max_gen_length
: Maximum number of tokens we set for each turn for the problem. The value is mostly 128 because each turn doesn't require substantial lines of code, but we adjusted a higher number when long generation is expected.{"prompts": ["Assign the string \"{A}\" to a variable named \"my_string\".",
"Lowercase the given string \"my_string\".",
"Assign the distinct characters of the string to a variable named \"chars\".",
"Sort these characters in alphabetical order.",
"Print the resulting list of characters." ],
"inputs": [ {"A": "abcde"},
{"A": "abcdecadeCADE"},
{"A": "aaaaAAAAaaaa"},
{"A": "Jerry jERRY JeRRRY"},
{"A": "ddddc"} ],
"outputs": [
["a", "b", "c", "d", "e"],
["a", "b", "c", "d", "e"],
["a"],
[" ", "e", "j", "r", "y"],
["c", "d"] ],
"max_gen_length": 128,
"category": "string",
"name": "Sandwich string",
"description": "Append a string in the middle of another string.",
"id": "1"}
table 4
fig 2
fig 3
fig 4
fig 5 (part)
fig 6(edited)
table 11
fig 3 (edited)
fig1![image](https://user-images.githubusercontent.com/36894403/234481545-931a88a7-e938-482a-869e-20516932dc64.png)