search

nus-apr / auto-code-rover

A project structure aware autonomous software engineer aiming for autonomous program improvement. Resolved 30.67% tasks (pass@1) in SWE-bench lite and 38.40% tasks (pass@1) in SWE-bench verified with each task costs less than $0.7.
Other
2.67k stars 276 forks source link

Why not generate the JSON with api calls directly in the first inference? #49

Open kripper opened 4 months ago

kripper commented 4 months ago

Is there a reason why you are doing 2 inferences instead of only 1 to analyze and generate the json with api calls? With a good prompt, we could do both together and save costs and noise. Do you want me to send a PR?

yuntongzhang commented 4 months ago

Currently we ask the model to only focus on analyzing first and ask it again to only focus on constructing json API calls. In our previous testing, we found this helps to generate higher quality API calls, compared to when the model analyzes and constructs API calls in one go.

However, if you have a good prompt that can generate high quality API calls in one request, that would be better since it's more cost-efficient. :)