search

princeton-nlp / SWE-agent

SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1 minute to run.
https://princeton-nlp.github.io/SWE-agent/
MIT License
13.29k stars 1.3k forks source link

Yanked packages? "Failed on sqlfluff__sqlfluff-1625: Failed to install requirements.txt" #707

Open samizdis opened 1 month ago

samizdis commented 1 month ago

Describe the bug

Many instances within SWE-Bench_Lite now fail (which previously succeeded), often pointing towards:

ERROR: Ignored the following yanked versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3                                                                                                                           
ERROR: Could not find a version that satisfies the requirement types-pkg_resources (from versions: none)                                                                                           
ERROR: No matching distribution found for types-pkg_resources

Maybe this is due to trying to install requirement.txt files which rely on packages which have since been yanked, but which weren't version-pinned?

I'm planning on looking into this more, but thought I'd start recording now.

Steps/commands/code to Reproduce

In a freshly cloned repo, run e.g. python run.py --model_name claude-sonnet-3.5 --instance_filter sqlfluff__sqlfluff-1625

(A full run against SWE-bench_Lite shows up many similarly-affected instances)

Error message/results

INFO     sqlfluff__sqlfluff__0.6 conda env not found, creating...                                                                                                                                           
DEBUG    Created conda environment                                                                                                                                                                          
ERROR    Failed to install requirements.txt: Collecting flake8 (from -r /root/requirements.txt (line 1))                                                                                                    
           Downloading flake8-7.1.1-py2.py3-none-any.whl.metadata (3.8 kB)                                                                                                                                  
         Collecting flake8-docstrings (from -r /root/requirements.txt (line 2))                                                                                                                             
           Downloading flake8_docstrings-1.7.0-py2.py3-none-any.whl.metadata (2.0 kB) 
           ...
           Collecting types-toml (from -r /root/requirements.txt (line 12))                                                                                                                                   
           Downloading types_toml-0.10.8.20240310-py3-none-any.whl.metadata (1.5 kB)                                                                                                                        
         ERROR: Ignored the following yanked versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3                                                                                                                           
         ERROR: Could not find a version that satisfies the requirement types-pkg_resources (from versions: none)                                                                                           
         ERROR: No matching distribution found for types-pkg_resources                                                                                                                                      
INFO     Beginning environment shutdown...                                                                                                                                                                  
INFO     Agent container stopped                                                                                                                                                                            
Traceback (most recent call last):
  File "/Users/sam/Repos/SWE-agent-2/run.py", line 381, in main
    self.run(index)
  File "/Users/sam/Repos/SWE-agent-2/run.py", line 345, in run
    observation, info = self.env.reset(index)
  File "/Users/sam/Repos/SWE-agent-2/sweagent/environment/swe_env.py", line 377, in reset
    self.install_env()
  File "/Users/sam/Repos/SWE-agent-2/sweagent/environment/swe_env.py", line 945, in install_env
    self.communicate_with_handling(
  File "/Users/sam/Repos/SWE-agent-2/sweagent/environment/swe_env.py", line 797, in communicate_with_handling
    raise RuntimeError(msg)
RuntimeError: Failed to install requirements.txt: Collecting flake8 (from -r /root/requirements.txt (line 1))
  Downloading flake8-7.1.1-py2.py3-none-any.whl.metadata (3.8 kB)
Collecting flake8-docstrings (from -r /root/requirements.txt (line 2))
  Downloading flake8_docstrings-1.7.0-py2.py3-none-any.whl.metadata (2.0 kB)
...

System Information

MacOS 12.7, Python 3.10.5, swe-agent main branch 36e430d27ffd11269738df92d6c521cab2207dcb

Checklist

klieret commented 1 month ago

I can confirm this on OSX

klieret commented 1 month ago

Also happens on Linux. Best way to reproduce is

python run.py --model_name instant_empty_submit --instance_filter sqlfluff__sqlfluff-1625 --noprint_config
klieret commented 1 month ago

There's probably not much to be done from the side of SWE-agent, because this is a SWE-bench thing, but I've reached out to Carlos + John to see if this also exists there.

klieret commented 1 month ago

It seems like the entire types-pkg-resources package was yanked (see https://pypi.org/project/types-pkg-resources/#history and https://pypi.org/project/types-pkg-resources/).

So this is not because an upper bound in requirements.txt. Perhaps we can just get away with pinning it to the latest version 0.1.3 everywhere?

klieret commented 1 month ago

Let's keep this open, but continue the discussion over at SWE-bench where the fix will happen: https://github.com/princeton-nlp/SWE-bench/issues/199