search

princeton-nlp / SWE-agent

SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1 minute to run.
https://princeton-nlp.github.io/SWE-agent/
MIT License
11.86k stars 1.18k forks source link

Can you help me out to reduce the cost? #622

Closed icoding1010 closed 4 days ago

icoding1010 commented 4 days ago

Describe the issue

Firstly, I appreciate the team for coming up this amazing open-source solution for GitHub issues. :)

Currently, I am using SWE-Agent for a use case. I observe that the cost is too much. Can you please answer the below two queries I have.

  1. Are the prompts(default_from_url.yml)+commands+demonstration sent to the LM at each step? Because I find that the total tokens (~8000 tokens) adds up in each step. By step, I mean the exchange between the LM and the terminal (PC). If yes, where do I change in the SWE-Agent code to prevent sending it to the LM at each step?

  2. What are some other ways you suggest to modify the SWE-Agent code to optimize the cost?

Thank you!

Optional: Relevant documentation page

No response

klieret commented 4 days ago

A significant part of the history (all inputs and outputs) are sent to the LM at every step. This is not avoidable (the LM does not have a memory of what it did before).

However, you could play around with compressing the history more (for example we strip out ouputs from commands from more than 5 steps ago).

But it's probably not gonna change the cost requirement drastically (or at least not without diminishing performance).

There's a few more things one could do when running on the same issue repeatedly or other tricks, but I'm pretty sure it will stay in the same order of magnitude, unfortunately...