Starting gateway container on VMs (16/16) in 50.64s
Exception in thread Thread-339:
Traceback (most recent call last):
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 1395, in getresponse
response.begin()
File "/pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 325, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 294, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/skyplane/api/tracker.py", line 153, in run
for chunk in chunk_stream:
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/skyplane/api/transfer_job.py", line 644, in dispatch
reply = self.http_pool.request(
^^^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/_request_methods.py", line 118, in request
return self.request_encode_body(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/_request_methods.py", line 217, in request_encode_body
return self.urlopen(method, url, **extra_kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/poolmanager.py", line 444, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 1395, in getresponse
response.begin()
File "/pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 325, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 294, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/pyenv/versions/3.11.9/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/skyplane/api/tracker.py", line 166, in run
UsageClient.log_exception(
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/skyplane/api/usage.py", line 147, in log_exception
stats = client.make_error(
^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/skyplane/api/usage.py", line 304, in make_error
dest_regions = [tag.split(":")[1] for tag in dest_region_tags]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.11.9/lib/python3.11/site-packages/skyplane/api/usage.py", line 304, in <listcomp>
dest_regions = [tag.split(":")[1] for tag in dest_region_tags]
~~~~~~~~~~~~~~^^^
IndexError: list index out of range
Dispatching chunks... 500 chunks (~6.38MB dispatched) 0/? bytes18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-022330e924095e459): SSH session not active
18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-04acb2f33cd98ed9a): SSH session not active
18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-08414153357859333): SSH session not active
18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-00de12d08c9130762): SSH session not active
18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-09853ce5b0234da47): SSH session not active
18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-03fa984063c2dac35): SSH session not active
18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-02cb36cebb176785b): SSH session not active
Dispatching chunks... 500 chunks (~6.38MB dispatched) 0/? bytes18:21:39 [ERROR] Error running copy_log, AWSServer(region_tag=aws:eu-north-1, instance_id=i-04833ba6f324caf25): SSH session not active
⠹ Dispatching chunks... 500 chunks (~6.38MB dispatched) 0/? bytes Traceback (most recent call last)
/pyenv/versions/3.11.9/lib/python3.11/site-packages/skyplane/cli/cli_transfer.py:511 in sync
matti
commented
2 months ago