search

dannyl1u / doppelganger

flags duplicate issues & PRs using embeddings
15 stars 1 forks source link

chore: setup swe-agent #33

Closed dannyl1u closed 1 day ago

amiicao commented 2 days ago

Tried swe-agent for #32 , a simple issue to use logging rather than print statements. Using OpenAI model GPT-4o mini, swe-agent was stuck in a loop for 3 hrs. It tried to implement test cases for the new code and was perpetually blocked by syntax errors.

amiicao commented 2 days ago

Considering this issue like a spike:

Using a pricier LLM API may yield better results, but a risk is a case where a loop occurs which incurs high costs without notice.

Edit: tried GPT4. Results are ~$2 spent and ~20 min runtime. I was unable to access the patch file due to Docker not loading, so there is no confirmation of working code. As many tokens are used to feed the code base into the LLM of choice, and code is outputted again and again until satisfactory, swe-agent is not economically scalable.

Thus, we will not be using swe-agent and I am closing this issue.