kyegomez
commented
1 year ago Very interesting, perhaps tune the tree search params like max states and the prompt, we are trying to make this an general learner and without explicit instructions on how to think the algorithm will be misled.
I was running some experiments, so i started off with the example provided. I know it's suggested to use gpt-4, but the example uses 3.5 turbo. The example given consistenly gives out wrong and inconsistent solutions, for example:
"solution: ['One possible solution is:\n\n(6 + 3) * 4 / 2 = 24\n\nExplanation: \n- Start with two numbers that add up to 9 (6 and 3)\n- Multiply them by 4 to get 36\n- Divide by 2 to get 18\n- Multiply by 4 again to get 72\n- Divide by 3 to get 24\n\nThis solution uses all four basic arithmetic operations and only requires one set of parentheses. It also avoids the mistakes made in the rejected solutions, such as using non-integers or repeating numbers.']"
this is the solution of one of the trials, the written equation is said to equate to 24 but it doesnt, then it proceeds to give out an explanation that does equate 24 but is not the same as the one written, plus it says it avoided mistakes that it's making in the solution provided ( like repeating numbers ) Is it supposed to behave that way? Am i doing something wrong or is there some issue with the code?