princeton-nlp / SWE-agent

SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1 minute to run.
https://princeton-nlp.github.io/SWE-agent/
MIT License
11.86k stars 1.18k forks source link

Issue running agent KeyError: 'module' #617

Closed ivan4722 closed 6 days ago

ivan4722 commented 6 days ago

Describe the bug

SWE-agent fails to produce output on one instance that I am trying to run, but it was working before ?

**edit I should mention that I tried to add some stuff to default.yaml, I tried adding Before you begin editing, first generate a dependency graph with the following code, you will get an adjacency list, and if necessary you may trace errors through the dependency graph. Make sure you create a python script with this dependency graph code at the root of the directory/repository that you are fixing. If you cannot generate the dependency graph after 3 tries, skip it. along with some python code to generate an adjacency list and this is what raises an error. Any idea why?

Steps/commands/code to Reproduce

python run.py --model_name gpt-3.5-turbo-0125 --instance_filter django__django-11001 --split test

Error message/results

INFO     💽 Loaded dataset from princeton-nlp/SWE-bench_Lite                                                                                                                                                    
INFO     Found image sweagent/swe-agent:latest with tags: ['sweagent/swe-agent:latest'], created: 2024-06-26T00:19:07.947463553Z for linux arm64.                                                               
DEBUG    Starting container with command: docker run -i --rm --name sweagent-swe-agent-latest-38e747dba0 sweagent/swe-agent:latest /bin/bash -l                                                                 
INFO     🌱 Environment Initialized                                                                                                                                                                             
DEBUG    Environment initialization took 7.00 seconds                                                                                                                                                           
INFO     Instance filter not matched. Skipping instance django__django-15202                                                                                                                                    
...                                                                                                                            
INFO     Instance filter not matched. Skipping instance sympy__sympy-16106                                                                                                                                      
INFO     ▶️  Beginning task 177                                                                                                                                                                                  
INFO     django__django not found in container, cloning...                                                                                                                                                      
DEBUG    Falling back to full cloning method due to multiple instances or persistent container                                                                                                                  
INFO     django__django__3.0 conda env not found, creating...                                                                                                                                                   
DEBUG    Created conda environment                                                                                                                                                                              
DEBUG    Installed requirements from requirements.txt                                                                                                                                                           
INFO     Installing django__django at base commit...                                                                                                                                                            
DEBUG    Ran install command                                                                                                                                                                                    
INFO     Installation step took 29.84 seconds                                                                                                                                                                   
INFO     Initializing agent settings for container e0c20d4e74fe2dbfb00c3082b2aa18af3bb1dbeaacd73981163b5e47a53a7271                                                                                             
INFO     Resetting model stats                                                                                                                                                                                  
INFO     SYSTEM (primary)                                                                                                                                                                                       
         SETTING: You are an autonomous programmer, and you're working directly in the command line with a special interface.                                                                                   

         The special interface consists of a file editor that shows you 100 lines of a file at a time.                                                                                                          
         In addition to typical bash commands, you can also use the following commands to help you navigate and edit files.                                                                                     

         COMMANDS:                                                                                                                                                                                              
         open:                                                                                                                                                                                                  
           docstring: opens the file at the given path in the editor. If line_number is provided, the window will be move to include that line                                                                  
           signature: open <path> [<line_number>]                                                                                                                                                               
           arguments:                                                                                                                                                                                           
             - path (string) [required]: the path to the file to open                                                                                                                                           
             - line_number (integer) [optional]: the line number to move the window to (if not provided, the window will start at the top of the file)                                                          

         goto:                                                                                                                                                                                                  
           docstring: moves the window to show <line_number>                                                                                                                                                    
           signature: goto <line_number>                                                                                                                                                                        
           arguments:                                                                                                                                                                                           
             - line_number (integer) [required]: the line number to move the window to                                                                                                                          

         scroll_down:                                                                                                                                                                                           
           docstring: moves the window down {WINDOW} lines                                                                                                                                                      
           signature: scroll_down                                                                                                                                                                               

         scroll_up:                                                                                                                                                                                             
           docstring: moves the window down {WINDOW} lines                                                                                                                                                      
           signature: scroll_up                                                                                                                                                                                 

         create:                                                                                                                                                                                                
           docstring: creates and opens a new file with the given name                                                                                                                                          
           signature: create <filename>                                                                                                                                                                         
           arguments:                                                                                                                                                                                           
             - filename (string) [required]: the name of the file to create                                                                                                                                     

         submit:                                                                                                                                                                                                
           docstring: submits your current code and terminates the session                                                                                                                                      
           signature: submit                                                                                                                                                                                    

         search_dir:                                                                                                                                                                                            
           docstring: searches for search_term in all files in dir. If dir is not provided, searches in the current directory                                                                                   
           signature: search_dir <search_term> [<dir>]                                                                                                                                                          
           arguments:                                                                                                                                                                                           
             - search_term (string) [required]: the term to search for                                                                                                                                          
             - dir (string) [optional]: the directory to search in (if not provided, searches in the current directory)                                                                                         

         search_file:                                                                                                                                                                                           
           docstring: searches for search_term in file. If file is not provided, searches in the current open file                                                                                              
           signature: search_file <search_term> [<file>]                                                                                                                                                        
           arguments:                                                                                                                                                                                           
             - search_term (string) [required]: the term to search for                                                                                                                                          
             - file (string) [optional]: the file to search in (if not provided, searches in the current open file)                                                                                             

         find_file:                                                                                                                                                                                             
           docstring: finds all files with the given name in dir. If dir is not provided, searches in the current directory                                                                                     
           signature: find_file <file_name> [<dir>]                                                                                                                                                             
           arguments:                                                                                                                                                                                           
             - file_name (string) [required]: the name of the file to search for                                                                                                                                
             - dir (string) [optional]: the directory to search in (if not provided, searches in the current directory)                                                                                         

         edit:                                                                                                                                                                                                  
           docstring: replaces lines <start_line> through <end_line> (inclusive) with the given text in the open file. The replacement text is terminated by a line with only end_of_edit on it. All of the     
         <replacement text> will be entered, so make sure your indentation is formatted properly. Python files will be checked for syntax errors after the edit. If the system detects a syntax error, the edit 
         will not be executed. Simply try to edit the file again, but make sure to read the error message and modify the edit command you issue accordingly. Issuing the same command a second time will just   
         lead to the same error message again.                                                                                                                                                                  
           signature: edit <start_line>:<end_line>                                                                                                                                                              
         <replacement_text>                                                                                                                                                                                     
         end_of_edit                                                                                                                                                                                            
           arguments:                                                                                                                                                                                           
             - start_line (integer) [required]: the line number to start the edit at                                                                                                                            
             - end_line (integer) [required]: the line number to end the edit at (inclusive)                                                                                                                    
             - replacement_text (string) [required]: the text to replace the current selection with                                                                                                             

         Please note that THE EDIT COMMAND REQUIRES PROPER INDENTATION.                                                                                                                                         
         If you'd like to add the line '        print(x)' you must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and  
         require fixing before it can be run.                                                                                                                                                                   

         RESPONSE FORMAT:                                                                                                                                                                                       
         Your shell prompt is formatted as follows:                                                                                                                                                             
         (Open file: <path>) <cwd> $                                                                                                                                                                            

         You need to format your output using two fields; discussion and command.                                                                                                                               
         Your output should always include _one_ discussion and _one_ command field EXACTLY as in the following example:                                                                                        
         DISCUSSION                                                                                                                                                                                             
         First I'll start by using ls to see what files are in the current directory. Then maybe we can look at some relevant files to see what they look like.                                                 
     ls -a                                                                                                                                                                                                  
     ```                                                                                                                                                                                                    

     You should only include a *SINGLE* command in the command section and then wait for a response from the shell before continuing with more discussion and commands. Everything you include in the       
     DISCUSSION section will be saved for future reference.                                                                                                                                                 
     If you'd like to issue two commands at once, PLEASE DO NOT DO THAT! Please instead first submit just the first command, and then after receiving a response you'll be able to issue the second command.
     You're free to use any other bash commands you want (e.g. find, grep, cat, ls, cd) in addition to the special commands listed above.                                                                   
     However, the environment does NOT support interactive session commands (e.g. python, vim), so please do not invoke them.                                                                               

INFO DEMONSTRATION:
/Users/ixiong/Desktop/csvtransfer/SWE-agent/trajectories/demonstrations/replaymarshmallow-codemarshmallow-1867default_sys-env_window100t-0.20p-0.95c-2.00install-1/marshmallow-codemars hmallow-1867.traj
INFO Trajectory will be saved to trajectories/ixiong/gpt-3.5-turbo-0125__SWE-bench_Litedefaultt-0.00p-0.95c-3.00install-1/djangodjango-11001.traj
Traceback (most recent call last): File "/Users/ixiong/Desktop/csvtransfer/SWE-agent/run.py", line 382, in main self.run(index) File "/Users/ixiong/Desktop/csvtransfer/SWE-agent/run.py", line 366, in run info, trajectory = self.agent.run( ^^^^^^^^^^^^^^^ File "/Users/ixiong/Desktop/csvtransfer/SWE-agent/sweagent/agent/agents.py", line 812, in run thought, action, output = self.forward(observation, env.get_available_actions(), state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/Desktop/csvtransfer/SWE-agent/sweagent/agent/agents.py", line 487, in forward thought, action, output = self.forward_with_error_check(observation, state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/Desktop/csvtransfer/SWE-agent/sweagent/agent/agents.py", line 645, in forward_with_error_check output = self.forward_model(observation, state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/Desktop/csvtransfer/SWE-agent/sweagent/agent/agents.py", line 532, in forward_model template.format( KeyError: 'module' WARNING ❌ Failed on djangodjango-11001: 'module'
INFO Beginning environment shutdown...
INFO Agent container stopped
INFO Found image sweagent/swe-agent:latest with tags: ['sweagent/swe-agent:latest'], created: 2024-06-26T00:19:07.947463553Z for linux arm64.
DEBUG Starting container with command: docker run -i --rm --name sweagent-swe-agent-latest-78298b9ef8 sweagent/swe-agent:latest /bin/bash -l
INFO 🌱 Environment Initialized
INFO Instance filter not matched. Skipping instance matplotlib
matplotlib-25433
....
INFO Instance filter not matched. Skipping instance django__django-12983



### System Information

MacOS M3 Python 3.12.3

### Checklist

- [X] I'm running with the latest docker container/on the latest development version (this means running both `git pull` and `docker pull sweagent/swe-agent:latest`)
- [X] I have copied the full command/code that I ran (as text, not as screenshot!)
- [X] If applicable: I have copied the **full** log file/error message that was the result (as text, not as screenshot!)
- [X] I have enclosed code/log messages in triple backticks ([docs](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#quoting-code)) and clicked "Preview" to make sure it's displayed correctly.
klieret commented 6 days ago

Sounds like you included {module} in one of your templates in config.yaml and there's no variable to fill this. Anything within braces is expected to be a variable.

ivan4722 commented 5 days ago

Sounds like you included {module} in one of your templates in config.yaml and there's no variable to fill this. Anything within braces is expected to be a variable.

Oh thanks, it runs now, but the agent seems to have trouble finding the script. I tried to mimic how you guys run "reproduce_bug.py", but it does not seem to work, here is the output:

(Open file: n/a)                                                                                                                                                                                                                            
         (Current directory: /django__django)                                                                                                                                                                                                        
         bash-$                                                                                                                                                                                                                                      
INFO     input_tokens=7,283, output_tokens=410, instance_cost=0.00, cost=0.00                                                                                                                                                                        
INFO     total_tokens_sent=7,283, total_tokens_received=410, total_cost=0.00, total_api_calls=1                                                                                                                                                      
INFO     💭 THOUGHT (primary)                                                                                                                                                                                                                        
         Let's start by following the instructions to generate the dependency graph for the repository. We need to create a Python script named `dependency_graph.py` in the root directory and paste the provided code into it. After that, we'll   
         run the script to generate the dependency graph.                                                                                                                                                                                            
     create dependency_graph.py                                                                                                                                                                                                                  
     ```                                                                                                                                                                                                                                         

     After creating the file, we need to edit it and paste the provided Python code for generating the dependency graph.                                                                                                                         

     ```                                                                                                                                                                                                                                         
     edit 1:1                                                                                                                                                                                                                                    
     import os                                                                                                                                                                                                                                   
     import ast                                                                                                                                                                                                                                  
     from collections import defaultdict                                                                                                                                                                                                         

     def find_python_files(directory):                                                                                                                                                                                                           
         python_files = []                                                                                                                                                                                                                       
         for root, _, files in os.walk(directory):                                                                                                                                                                                               
             for file in files:                                                                                                                                                                                                                  
                 if file.endswith(".py"):                                                                                                                                                                                                        
                     python_files.append(os.path.join(root, file))                                                                                                                                                                               
         return python_files                                                                                                                                                                                                                     

     def extract_imports(file_path):                                                                                                                                                                                                             
         with open(file_path, "r") as file:                                                                                                                                                                                                      
             tree = ast.parse(file.read(), filename=file_path)                                                                                                                                                                                   

         imports = []                                                                                                                                                                                                                            
         for node in ast.walk(tree):                                                                                                                                                                                                             
             if isinstance(node, ast.Import):                                                                                                                                                                                                    
                 for alias in node.names:                                                                                                                                                                                                        
                     imports.append(alias.name)                                                                                                                                                                                                  
             elif isinstance(node, ast.ImportFrom):                                                                                                                                                                                              
                 module = node.module                                                                                                                                                                                                            
                 for alias in node.names:                                                                                                                                                                                                        
                     if module:                                                                                                                                                                                                                  
                         imports.append(module + "." + alias.name)                                                                                                                                                                               
                     else:                                                                                                                                                                                                                       
                         imports.append(alias.name)                                                                                                                                                                                              
         return imports                                                                                                                                                                                                                          

     def create_dependency_graph(directory):                                                                                                                                                                                                     
         python_files = find_python_files(directory)                                                                                                                                                                                             
         dependency_graph = defaultdict(list)                                                                                                                                                                                                    

         for file_path in python_files:                                                                                                                                                                                                          
             file_name = os.path.relpath(file_path, directory)                                                                                                                                                                                   
             imports = extract_imports(file_path)                                                                                                                                                                                                
             for imp in imports:                                                                                                                                                                                                                 
                 dependency_graph[file_name].append(imp)                                                                                                                                                                                         

         return dependency_graph                                                                                                                                                                                                                 

     repo_directory = os.path.dirname(os.path.abspath(__file__))                                                                                                                                                                                 
     dependency_graph = create_dependency_graph(repo_directory)                                                                                                                                                                                  
     for file, dependencies in dependency_graph.items():                                                                                                                                                                                         
         print(file + ": " + str(dependencies))                                                                                                                                                                                                  
     end_of_edit                                                                                                                                                                                                                                 
     ```                                                                                                                                                                                                                                         

     Now, let's run the script to generate the dependency graph.                                                                                                                                                                                 

INFO 🎬 ACTION (primary)
python dependency_graph.py

INFO 🤖 MODEL INPUT
python: can't open file 'dependency_graph.py': [Errno 2] No such file or directory
(Open file: n/a)
(Current directory: /django__django)
bash-$
INFO input_tokens=7,741, output_tokens=48, instance_cost=0.01, cost=0.00
INFO total_tokens_sent=15,024, total_tokens_received=458, total_cost=0.01, total_api_calls=2
INFO 💭 THOUGHT (primary)
It seems like there was an issue with running the Python script because the file dependency_graph.py was not found. Let's first ensure that the file was created successfully by listing the files in the current directory.

Here is the prompt: 

Before you begin editing, we recommend generating a dependency graph in your environment (code is provided). You will get an adjacency list, and if necessary, you may trace errors through the dependency graph.

  Retries:
  If you cannot generate the dependency graph after three tries, skip this step.

Do you have any advice on how to fix this?