search

NREL / rex

REsource eXtraction Tool (rex)
https://nrel.github.io/rex
BSD 3-Clause "New" or "Revised" License
19 stars 10 forks source link

connection refused issues on eagle #93

Closed tarekelgindy closed 3 years ago

tarekelgindy commented 3 years ago

Hi there

I recently tried to run the following python code on eagle:

nsrdb_file = '/datasets/NSRDB/v3/nsrdb_2018.h5'
from rex import NSRDBX
nrel = (39.741931, -105.169891)
with NSRDBX(nsrdb_file, hsds=True) as f:
    nrel_dni = f.get_lat_lon_df('dni', nrel)

And got the following warning messages, which eventually timed out with the full traceback:

WARNING:urllib3.connectionpool:Retrying (Retry(total=9, connect=9, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea7240>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=8, connect=8, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea7518>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=7, connect=7, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea76a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=6, connect=6, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea7828>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=5, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea79b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=4, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea7b38>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=3, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea7cf8>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ea7eb8>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ec80b8>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=10, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ec8278>: Failed to establish a new connection: [Errno 111] Connection refused')': /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5
ERROR:root:connection error: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ec8438>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/telgindy/miniconda3/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/telgindy/miniconda3/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/telgindy/miniconda3/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/telgindy/miniconda3/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/home/telgindy/miniconda3/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7feaa3ec8438>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  [Previous line repeated 7 more times]
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ec8438>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/h5pyd/_hl/httpconn.py", line 344, in GET
    rsp = s.get(self._endpoint + req, params=params, headers=headers, verify=self.verifyCert())
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/h5pyd/_hl/httpconn.py", line 48, in send
    return super().send(request, **kwargs)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /?getdnids=1&domain=%2Fdatasets%2FNSRDB%2Fv3%2Fnsrdb_2018.h5 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7feaa3ec8438>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/rex/resource_extraction/resource_extraction.py", line 1274, in __init__
    res_cls=NSRDB)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/rex/resource_extraction/resource_extraction.py", line 63, in __init__
    str_decode=str_decode, group=group)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/rex/resource.py", line 562, in __init__
    self._h5 = h5pyd.File(self.h5_file, mode='r', use_cache=False)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/h5pyd/_hl/files.py", line 229, in __init__
    rsp = http_conn.GET(req, params=params)
  File "/home/telgindy/miniconda3/lib/python3.7/site-packages/h5pyd/_hl/httpconn.py", line 348, in GET
    raise IOError("Connection Error")
OSError: Connection Error

I think there might be some firewall issues that are causing this.

However, tracing the error, I found that the culprit was located here:

By replacing this line with

self._h5 = h5py.File(self.h5_file, mode='r')

This problem was able to be avoided. I think this won't be compatible with the http requests that might be made to AWS, but it might be worth checking if local files are being used and using the simpler h5py calls if that's the case.

MRossol commented 3 years ago

@tarekelgindy, this isn't a bug, the hsds=True flag is to access data on AWS via our HSDS service, BUT the file path your providing is local on Eagle.