How do I inference and evaluate on SWE-Bench faster by reusing previous built env?

[Yuzz1020]

Describe the issue

Hi,

Thank you for releasing this awesome repo! I've been enjoying reading and trying it out.

I'm trying to run it on the SWE-Bench-lite dataset and found that it takes hours to finish the generation process so I want to run multiple questions in SWE-Bench in parallel to speed it up. However, when I'm doing so (basically running multiple run.py at the same time), I got the following error when creating environment:

ERROR    Read with timeout failed on input:
         ---
         /bin/bash -n <<'EOF'
         source /root/.bashrc
         EOF

         ---
Traceback (most recent call last):
  File "/home/ubuntu/SWE-agent/run.py", line 377, in main
    self.run(index)
  File "/home/ubuntu/SWE-agent/run.py", line 341, in run
    observation, info = self.env.reset(index)
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 328, in reset
    self.communicate_with_handling(
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 722, in communicate_with_handling
    logs = self.communicate(input, timeout_duration=timeout_duration)
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 698, in communicate
    output = self._communicate(
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 666, in _communicate
    raise e
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 663, in _communicate
    exit_code = read_with_timeout(self.container, self.get_pids, 5).strip()
  File "/home/ubuntu/SWE-agent/sweagent/environment/utils.py", line 177, in read_with_timeout
    raise TimeoutError(msg)
TimeoutError: Timeout reached while reading from subprocess.
Current buffer: 0

Running PIDs: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/SWE-agent/run.py", line 516, in <module>
    main(get_args())
  File "/home/ubuntu/SWE-agent/run.py", line 512, in main
    Main(args).main()
  File "/home/ubuntu/SWE-agent/run.py", line 398, in main
    self.env.reset_container()
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 538, in reset_container
    self._reset_container()
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 532, in _reset_container
    self._init_scripts()
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 598, in _init_scripts
    self.communicate_with_handling(
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 722, in communicate_with_handling
    logs = self.communicate(input, timeout_duration=timeout_duration)
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 695, in communicate
    output, valid = self._check_syntax(input)
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 677, in _check_syntax
    output = self._communicate(f"/bin/bash -n <<'EOF'\n{input}\nEOF\n")
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 666, in _communicate
    raise e
  File "/home/ubuntu/SWE-agent/sweagent/environment/swe_env.py", line 663, in _communicate
    exit_code = read_with_timeout(self.container, self.get_pids, 5).strip()
  File "/home/ubuntu/SWE-agent/sweagent/environment/utils.py", line 177, in read_with_timeout
    raise TimeoutError(msg)
TimeoutError: Timeout reached while reading from subprocess.
Current buffer: 0

Running PIDs: []

Moreover, I noticed that the major bottleneck in efficiency is the time needed to clone and build the conda environment. This process seems to be required every time I do an evaluation, even I've already built the environment last time. Thus, I am also wondering if there is a way to keep the conda environment I built previously to save some time.

Thank you for your help in advance!

Optional: Relevant documentation page

No response

[klieret]

Hi @Yuzz1020 . A couple of things:

1. It seems like you're probably not on the newest version? We've introduced major speedups in the last couple of days (changelog)
2. We now have a speedup section in the docs

The tl;dr of 2 is probably to use --container_name my_container_name (use a different one if you run multiple instances of run.py). This will make SWE-agent keep working in the same container and reuse cloned repos and installed conda environments.

I actually also wonder if we have a SWE-agent container with all the SWE-bench repos pre-installed. Let me ping @carlosejimenez @john-b-yang here.

[Yuzz1020]

Thank you for your response, it is really helpful! It solved my issues for now.

Also, I do agree that it would be awesome if there can be a container with all SWE-bench repos pre-installed!