Download issues

[falkamelung]

ssara_federated_query-cj.py:

*******************  FIRST COOKIE Check  ******************
 > attempting to download https://urs.earthdata.nasa.gov/profile
Exception in thread Thread-7:
Traceback (most recent call last):
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/nethome/famelung/test/dev/rsmas_insar/3rdparty/miniconda3/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

[mirzaees]

@falkamelung , is this still an issue?

[falkamelung]

FROM TACC CONSULT

1. Generally, reducing the transfers to as few as possible and avoiding small transfers will be the least taxing for the system. Parallel transfers can occupy an disproportionate amount of resources.

2. Using /scratch would be preferred, but the transfers should be relatively similar. The precaution is to avoid quickly filling up the very limited user storage space in $HOME.

3/4. Stampede's six login nodes each have two 10GigE ethernet adapters that are channel bonded but the 5936 compute nodes each has a single 1GigE adapter.

I hope the clears things up well enough, it's difficult to answer based on files but you can use that hardware and network information to gauge how best to approach.