Feature request: Create a PR on the repo

[catid]

I gave it a github api token that has access to my playground repo: https://github.com/catid/swe_agent_playground

Token is in the keys.cfg but it doesn't seem to read that so I also manually exported the key envvars.

It completed the task but did not create a PR or branch and does not have any useful errors explaining why:

         Let me know if you would like me to explain or modify anything! Otherwise, I think we can go
         ahead and submit this as the solution.

INFO     🎬 ACTION (primary)
         submit

INFO     Found submission:

INFO     Saved trajectory to
         trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c-
         2.00__install-1/catid__swe_agent_playground-i1.traj
INFO     Saved predictions to
         trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c-
         2.00__install-1/all_preds.jsonl
(swe-agent) ➜  SWE-agent git:(main)

Also no useful logs explaining why it did not submit anything in the trajectories folder.

Also noted that this project fails to start if the repo is empty. I'd prefer to have it work on empty repos too.

[joskfg]

Same happens to me: https://github.com/princeton-nlp/SWE-agent/issues/4#issuecomment-2032853966

[timothycarambat]

This repo does not automatically interact with the github repo/issue that you are running it on. It generates a diff which can then be applied to a repo via git, but it is not an automated tool that manages repos. If that was the case every test in the SWE-bench lite would have thousands of PRS from people running this tool and others like it.

In trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c- 2.00__install-1/catid__swe_agent_playground-i1.traj you can see this is a JSON file. Scroll to the bottom. You are looking for info and a sub-key called submission.

you can then apply the patch manually that was proposed

[ofirpress]

Timothy is correct. Although it could be cool to make this as an optional flag for private, non SWE-bench repos.

[cocobeach]

ok so the repo needs to be Public? I might have similar problem...

I have created an issue on my repo and used this cmd: python run.py --model_name claude-3-opus-20240229 \ --data_path https://github.com/cocobeach/Killer-Wasp/issues/1 --config_file config/default_from_url.yaml

(where is tha config_file?)

///ERROR: 

Traceback (most recent call last):
  File "/home/maxtensor/Dev/SWE-agent/run.py", line 223, in <module>
    main(args)
  File "/home/maxtensor/Dev/SWE-agent/run.py", line 66, in main
    env = SWEEnv(args.environment)
  File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/swe_env.py", line 88, in __init__
    self.data = get_instances(self.data_path, self.args.base_commit, self.args.split)
  File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/utils.py", line 302, in get_instances
    issue = api.issues.get(owner, repo, issue_number)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 62, in __call__
    return self.client(self.path, self.verb, headers=headers, route=route_p, query=query_p, data=data_p)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 121, in __call__
    res,self.recv_hdrs = urlsend(path, verb, headers=headers or None, debug=debug, return_headers=True,
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 218, in urlsend
    return urlread(req, return_json=return_json, return_headers=return_headers)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 119, in urlread
    if 400 <= e.code < 500: raise ExceptionsHTTP[e.code](e.url, e.hdrs, e.fp, msg=e.msg) from None
fastcore.net.HTTP404NotFoundError: HTTP Error 404: Not Found
====Error Body====
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/issues/issues#get-an-issue"
}

Any ideas?

Thanks a lot Great Repo, we all love it!!

[farouqaldori]

Right now the submit function only displays the --diff as far as I know, maybe a simple solution would be to create the patch and push it to the repo?

defaults.sh

# @yaml
# signature: submit
# docstring: submits your current code and terminates the session
submit() {
    cd $ROOT

    # Check if the patch file exists and is non-empty
    if [ -s "/root/test.patch" ]; then
        # Apply the patch in reverse
        git apply -R < "/root/test.patch"
    fi

    git add -A
    git diff --cached > model.patch
    echo "<<SUBMISSION||"
    cat model.patch
    echo "||SUBMISSION>>"
}

[timothycarambat]

@cocobeach No, it does not matter if the repo is public, private, or anything. The functionality to automatically push the diff to Github simply does not exist. This repo does not do that. If you wanted it to do that though, you can add it as a function the agent can all.

[cocobeach]

Ok then I don't understand, what happens with the issue number we give it when we hit run.py, what is it supposed to do in theory? Where are we supposed to retrieve the answer?

[klieret]

Ok then I don't understand, what happens with the issue number we give it when we hit run.py, what is it supposed to do in theory? Where are we supposed to retrieve the answer?

The scope of the project so far was to demonstrate and quantify the ability to solve real world gh issues. As @farouqaldori correctly pointed out, you can see the git diff and semi-manually apply it.

I will look into whether we can add PR functionality soon.

[gcohen1928]

ok so the repo needs to be Public? I might have similar problem...

I have created an issue on my repo and used this cmd: python run.py --model_name claude-3-opus-20240229 \ --data_path https://github.com/cocobeach/Killer-Wasp/issues/1 --config_file config/default_from_url.yaml

(where is tha config_file?)

///ERROR: 

Traceback (most recent call last):
  File "/home/maxtensor/Dev/SWE-agent/run.py", line 223, in <module>
    main(args)
  File "/home/maxtensor/Dev/SWE-agent/run.py", line 66, in main
    env = SWEEnv(args.environment)
  File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/swe_env.py", line 88, in __init__
    self.data = get_instances(self.data_path, self.args.base_commit, self.args.split)
  File "/home/maxtensor/Dev/SWE-agent/sweagent/environment/utils.py", line 302, in get_instances
    issue = api.issues.get(owner, repo, issue_number)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 62, in __call__
    return self.client(self.path, self.verb, headers=headers, route=route_p, query=query_p, data=data_p)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/ghapi/core.py", line 121, in __call__
    res,self.recv_hdrs = urlsend(path, verb, headers=headers or None, debug=debug, return_headers=True,
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 218, in urlsend
    return urlread(req, return_json=return_json, return_headers=return_headers)
  File "/home/maxtensor/Dev/SWE-agent/.venv/lib/python3.9/site-packages/fastcore/net.py", line 119, in urlread
    if 400 <= e.code < 500: raise ExceptionsHTTP[e.code](e.url, e.hdrs, e.fp, msg=e.msg) from None
fastcore.net.HTTP404NotFoundError: HTTP Error 404: Not Found
====Error Body====
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/issues/issues#get-an-issue"
}

Any ideas?

Thanks a lot Great Repo, we all love it!!

Had the same error where it couldnt access my private repo but then added my github token to my bash profile and then it worked. @cocobeach

[klieret]

As also reported in #20 , the GITHUB_TOKEN was not read from keys.cfg so far, so you had to specify it as an environment variable. This will be fixed with #31 :)