search

EGA-archive / ega-download-client

A Python-based EGA download client
Apache License 2.0
94 stars 52 forks source link

Alternative to pyega3 ? #200

Closed PFRoux closed 1 year ago

PFRoux commented 1 year ago

Dear all,

I've been trying to use pyega3 on 2 different academic computational clusters and my own laptop (macOS) for more than a week and I've been through all possible issues so far : pyega3 incompatible with the cluster architecture, connection problem, connection broken, md5 check inconsistencies ... and haven't been able to download one single file properly. I am losing my mind !

So far, I am still waiting for the support to answer my inquiries. But the thing is that I need to download the data I am supposed to have access to quickly, since one of the master student I am supervising needs it to finish is thesis (ending at the end of the month ...).

Is there an alternative to this download client ?

Thanks a lot for your help.

Pierre-François

CsabaHalmagyi commented 1 year ago

Dear Pierre-Francois, Thank you for contacting us. I would like to confirm that the EGA development team is investigating the downloading issues. For further assistance, please email the EGA Helpdesk at helpdesk@ega-archive.org, they can provide alternative ways to download the datasets. Regards, Csaba

JianGuoZhou3 commented 1 year ago

[2023-07-30 05:06:47 -0400] [2023-07-30 05:06:47 -0400] pyEGA3 - EGA python client version 5.0.2 (https://github.com/EGA-archive/ega-download-client) [2023-07-30 05:06:47 -0400] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly [2023-07-30 05:06:47 -0400] Python version : 3.11.4 [2023-07-30 05:06:47 -0400] OS version : Linux #1 SMP Debian 5.10.179-3 (2023-07-27) [2023-07-30 05:06:47 -0400] Server URL: https://ega.ebi.ac.uk:8443/v2 [2023-07-30 05:06:47 -0400] Session-Id: 2861361300 [2023-07-30 05:06:48 -0400] [2023-07-30 05:06:48 -0400] Authentication success for user 'jianguo.zhou@zmu.edu.cn' [2023-07-30 05:06:50 -0400] File Id: 'EGAF00005292867'(25222818 bytes). [2023-07-30 05:06:50 -0400] Total space : 1692.88 GiB [2023-07-30 05:06:50 -0400] Used space : 3.13 GiB [2023-07-30 05:06:50 -0400] Free space : 1620.93 GiB [2023-07-30 05:06:50 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]... [2023-07-30 05:18:56 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 7 more times] File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry self.download_file(output_file, num_connections, max_slice_size) File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file for part_file_name in executor.map(self.download_fileslice, params): File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator yield _result_or_cancel(fs.pop()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel return fut.result(timeout) ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_fileslice return self.download_file_slice(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice with self.data_client.get_stream(path, extra_headers) as r: File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter return next(self.gen) ^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) [2023-07-30 05:19:56 -0400] retry attempt 1 [2023-07-30 05:19:56 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]... [2023-07-30 05:32:01 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 7 more times] File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry self.download_file(output_file, num_connections, max_slice_size) File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file for part_file_name in executor.map(self.download_fileslice, params): File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator yield _result_or_cancel(fs.pop()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel return fut.result(timeout) ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_fileslice return self.download_file_slice(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice with self.data_client.get_stream(path, extra_headers) as r: File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter return next(self.gen) ^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) [2023-07-30 05:33:01 -0400] retry attempt 2 [2023-07-30 05:33:01 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]... [2023-07-30 05:45:07 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 7 more times] File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry self.download_file(output_file, num_connections, max_slice_size) File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file for part_file_name in executor.map(self.download_fileslice, params): File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator yield _result_or_cancel(fs.pop()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel return fut.result(timeout) ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_fileslice return self.download_file_slice(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice with self.data_client.get_stream(path, extra_headers) as r: File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter return next(self.gen) ^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) [2023-07-30 05:46:07 -0400] retry attempt 3 [2023-07-30 05:46:07 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]... [2023-07-30 05:58:12 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 7 more times] File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry self.download_file(output_file, num_connections, max_slice_size) File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file for part_file_name in executor.map(self.download_fileslice, params): File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator yield _result_or_cancel(fs.pop()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel return fut.result(timeout) ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_fileslice return self.download_file_slice(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice with self.data_client.get_stream(path, extra_headers) as r: File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter return next(self.gen) ^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) [2023-07-30 05:59:12 -0400] retry attempt 4 [2023-07-30 05:59:12 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]... [2023-07-30 06:06:37 -0400] [2023-07-30 06:06:37 -0400] pyEGA3 - EGA python client version 5.0.2 (https://github.com/EGA-archive/ega-download-client) [2023-07-30 06:06:37 -0400] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly [2023-07-30 06:06:37 -0400] Python version : 3.11.4 [2023-07-30 06:06:37 -0400] OS version : Linux #1 SMP Debian 5.10.179-3 (2023-07-27) [2023-07-30 06:06:37 -0400] Server URL: https://ega.ebi.ac.uk:8443/v2 [2023-07-30 06:06:37 -0400] Session-Id: 1480591647 [2023-07-30 06:06:39 -0400] [2023-07-30 06:06:39 -0400] Authentication success for user 'jianguo.zhou@zmu.edu.cn' [2023-07-30 06:06:41 -0400] File Id: 'EGAF00005292867'(25222818 bytes). [2023-07-30 06:06:41 -0400] Total space : 1692.88 GiB [2023-07-30 06:06:41 -0400] Used space : 3.13 GiB [2023-07-30 06:06:41 -0400] Free space : 1620.93 GiB [2023-07-30 06:06:41 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]... [2023-07-30 06:11:18 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses')) urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

JianGuoZhou3 commented 1 year ago

I still can't download the EGA datasets... I can confirm my internet is fine. Jian-Guo image

@CsabaHalmagyi

CsabaHalmagyi commented 1 year ago

Dear @jianguozhouzunyimedicaluniversity Thank you for submitting the error log. We looked into the report and identified the root cause issue. Could you please retry downloading the dataset(s)? Should you encounter any issues please contact our helpdesk team at helpdesk@ega-archive.org, they can provide alternative ways to download the datasets. Regards, Csaba