Closed nicovank closed 1 year ago
It seems to do well with the Python example, failures coming from indentation (also the reason for a lot of 2 retries instead of 1). Maybe we can do something about that.
% python3 tests/runner-diff.py Python concatenate-str-to-int.py
============================ CWhy Test Runner ============================
LLM : gpt-4
Timeout : 180
Iterations : 10
Max retries : 5
Benchmark : Python/concatenate-str-to-int.py
==========================================================================
Trying to repair concatenate-str-to-int.py (/tmp/tmpnvd_pwwh)...
Success in 2 retries! Total tokens used per retry: [321, 222]
Trying to repair concatenate-str-to-int.py (/tmp/tmp1g7zjm1w)...
Success in 2 retries! Total tokens used per retry: [329, 226]
Trying to repair concatenate-str-to-int.py (/tmp/tmp1zmrclxw)...
Failed after 5 retries. Total tokens used per retry: [325, 224, 211, 222, 222]
Trying to repair concatenate-str-to-int.py (/tmp/tmpjwsobezo)...
Success in 2 retries! Total tokens used per retry: [321, 222]
Trying to repair concatenate-str-to-int.py (/tmp/tmp1vykp2tb)...
Success in 2 retries! Total tokens used per retry: [321, 222]
Trying to repair concatenate-str-to-int.py (/tmp/tmpnsfmy39g)...
Success in 2 retries! Total tokens used per retry: [321, 222]
Trying to repair concatenate-str-to-int.py (/tmp/tmpbtprjz36)...
Success in 2 retries! Total tokens used per retry: [323, 226]
Trying to repair concatenate-str-to-int.py (/tmp/tmpv95awox3)...
Success in 2 retries! Total tokens used per retry: [321, 222]
Trying to repair concatenate-str-to-int.py (/tmp/tmpytv0fs6r)...
Success in 2 retries! Total tokens used per retry: [327, 228]
Trying to repair concatenate-str-to-int.py (/tmp/tmpz8ze9xf3)...
Success in 2 retries! Total tokens used per retry: [325, 224]
Python/concatenate-str-to-int.py success rate: 90.00%
This adds a diff function, which returns proposed changes in JSON format. In the example below, GPT suggested a single change, a second request is likely to push it to also give a
pair_hash
implementation.It also adds a test runner for this new command, that essentially tries to apply modifications recursively until the code compiles or some max tree depth is reached. Successive errors are not kept in context: each GPT request is sent as if it was the first one.
The current max retries default is 5, I've seen successful runs in up to 4 retries (3 retries in sample output above). It is typically pretty clear for the human eye when GPT is lost for a given tree, not sure if there is any way to detect these cases and exit early.
For the missing-hash example that I've been experimenting with, failures typically happen because: