ivan4722
commented
3 months ago Nevermind, docker desktop just wasn't open it seems. To confirm, is this the expected output?
(base) nb24-12252:SWE-bench ixiong$ python -m swebench.harness.run_evaluation --predictions_path gold --max_workers 1 --instance_ids sympy__sympy-20590 --run_id validate-gold
<frozen runpy>:128: RuntimeWarning: 'swebench.harness.run_evaluation' found in sys.modules after import of package 'swebench.harness', but prior to execution of 'swebench.harness.run_evaluation'; this may result in unpredictable behaviour
Using gold predictions - ignoring predictions_path
Running 1 unevaluated instances...
Building base image (sweb.base.x86_64:latest)
Base images built successfully.
Total environment images to build: 1
Building environment images: 100%|██████████████████████████████████████████████████████████████| 1/1 [00:46<00:00, 46.40s/it]
All environment images built successfully.
Running 1 instances...
100%|███████████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:17<00:00, 77.14s/it]
All instances run.
Cleaning cached images...
Removed 0 images.
Total instances: 1
Instances completed: 1
Instances resolved: 1
Instances unresolved: 0
Instances with errors: 0
Instances still running: 0
Still existing images: 0
Report written to gold.validate-gold.json
john-b-yang
Describe the bug
I followed the exact steps in the setup but I cannot run the command
Steps/Code to Reproduce
Expected Results
No error
Actual Results
(base) nb24-12252:SWE-bench ixiong$ python -m swebench.harness.run_evaluation \
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 844, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise raise value.with_traceback(tb) File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 496, in _make_request conn.request( File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/Users/ixiong/miniconda3/lib/python3.12/http/client.py", line 1331, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/Users/ixiong/miniconda3/lib/python3.12/http/client.py", line 1091, in _send_output self.send(msg) File "/Users/ixiong/miniconda3/lib/python3.12/http/client.py", line 1035, in send self.connect() File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/api/client.py", line 223, in _retrieve_server_version return self.version(api_version=False)["ApiVersion"] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version return self._result(self._get(url), json=True) ^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/utils/decorators.py", line 44, in inner return f(self, *args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/api/client.py", line 246, in _get return self.get(url, self._set_request_timeout(kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/requests/adapters.py", line 682, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/Users/ixiong/Desktop/SWE-bench/swebench/harness/run_evaluation.py", line 529, in
main(*vars(args))
File "/Users/ixiong/Desktop/SWE-bench/swebench/harness/run_evaluation.py", line 466, in main
client = docker.from_env()
^^^^^^^^^^^^^^^^^
File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/client.py", line 94, in from_env
return cls(
^^^^
File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/client.py", line 45, in init
self.api = APIClient(args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/api/client.py", line 207, in init
self._version = self._retrieve_server_version()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ixiong/miniconda3/lib/python3.12/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
System Information
MAC OS, I just cloned swebench (2.0.1) python 3.12.3