Skip tasks that fail setup (rather than crashing)

When harness/run_setup.py fails for a task we get a crash rather than a helpful message.

Also I think it would be better if tasks that fail setup are skipped rather than causing the whole run to fail.

Here is the failure that occurs because Miniconda on arm64 doesn't support Python 3.6:

(auto-code-rover) root@2ae90efb13f1:/opt/auto-code-rover# cd /opt/SWE-bench
echo django__django-11133 > tasks.txt
(auto-code-rover) root@2ae90efb13f1:/opt/SWE-bench# cd /opt/SWE-bench
conda activate swe-bench
python harness/run_setup.py --log_dir logs --testbed testbed --result_dir setup_result --subset_file tasks.txt
2024-04-15 03:05:28,024 - INFO - env_name for all setup entries: ['setup_django__django__3.0']
2024-04-15 03:05:28,024 - INFO - Starting parallel setup.
2024-04-15 03:05:28,024 - INFO -    Number of setup tasks: 1
2024-04-15 03:05:28,024 - INFO -    Number of processes: 1
2024-04-15 03:05:28,024 - INFO - [setup_django__django__3.0] ======= Start setting up for django/django 3.0 =======
2024-04-15 03:07:48,933 - INFO - [setup_django__django__3.0] Cloned django/django to /opt/SWE-bench/testbed/django__django/setup_django__django__3.0
2024-04-15 03:07:49,092 - INFO - [setup_django__django__3.0] Creating environment setup_django__django__3.0; Command: /opt/conda/bin/conda create -n setup_django__django__3.0 python=3.6 -y
2024-04-15 03:07:50,584 - ERROR - Error: Command '['/opt/conda/bin/conda', 'create', '-n', 'setup_django__django__3.0', 'python=3.6', '-y']' returned non-zero exit status 1.
2024-04-15 03:07:50,585 - ERROR - Error stdout: Channels:
 - defaults
Platform: linux-aarch64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

2024-04-15 03:07:50,585 - ERROR - Error stderr: 
PackagesNotFoundError: The following packages are not available from current channels:

  - python=3.6*

Current channels:

  - defaults

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

2024-04-15 03:07:50,587 - ERROR - Error traceback: Traceback (most recent call last):
  File "/opt/SWE-bench/harness/context_manager.py", line 49, in __call__
    output = subprocess.run(cmd, **combined_args)
  File "/opt/conda/envs/swe-bench/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/conda/bin/conda', 'create', '-n', 'setup_django__django__3.0', 'python=3.6', '-y']' returned non-zero exit status 1.

Done with setup.
setup_map is saved to setup_result/setup_map.json
tasks_map is saved to setup_result/tasks_map.json
Traceback (most recent call last):
  File "/opt/SWE-bench/harness/run_setup.py", line 440, in <module>
    main(**vars(args))
  File "/opt/SWE-bench/harness/run_setup.py", line 382, in main
    setup_one_repo_version(*entry)
  File "/opt/SWE-bench/harness/run_setup.py", line 233, in setup_one_repo_version
    create_conda_env(repo_full, version, repo_path, env_name, task)
  File "/opt/SWE-bench/harness/run_setup.py", line 132, in create_conda_env
    exec_wrapper(cmd.split(" "))
  File "/opt/SWE-bench/harness/context_manager.py", line 57, in __call__
    raise e
  File "/opt/SWE-bench/harness/context_manager.py", line 49, in __call__
    output = subprocess.run(cmd, **combined_args)
  File "/opt/conda/envs/swe-bench/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/conda/bin/conda', 'create', '-n', 'setup_django__django__3.0', 'python=3.6', '-y']' returned non-zero exit status 1.

I plan to submit a suitable PR shortly.

yuntongzhang / SWE-bench

Skip tasks that fail setup (rather than crashing) #2