search

LCOGT / neoexchange

NEO observing portal
GNU General Public License v3.0
7 stars 1 forks source link

Further connection issues with HORIZONS #495

Closed talister closed 2 years ago

talister commented 3 years ago

View details in Rollbar: https://rollbar.com/LCO/NEOexchange/items/177/

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 179, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/www/apps/neoexchange/core/plots.py", line 175, in make_visibility_plot
    ephem = horizons_ephem(body.name, start, end, site_code, include_moon=True)
  File "/var/www/apps/neoexchange/astrometrics/ephem_subs.py", line 656, in horizons_ephem
    max_hour_angle=ha_limit)
  File "/usr/local/lib/python3.6/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
    return self.fn(obj, *args, **kwds)
  File "/usr/local/lib/python3.6/site-packages/astroquery/utils/process_asyncs.py", line 26, in newmethod
    response = getattr(self, async_method_name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/astroquery/jplhorizons/core.py", line 599, in ephemerides_async
    timeout=self.TIMEOUT, cache=cache)
  File "/usr/local/lib/python3.6/site-packages/astroquery/query.py", line 263, in _request
    json=json)
  File "/usr/local/lib/python3.6/site-packages/astroquery/query.py", line 71, in request
    json=json)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='ssd.jpl.nasa.gov', port=443): Max retries exceeded with url: /horizons_batch.cgi?batch=1&TABLE_TYPE=OBSERVER&QUANTITIES=%271%2C3%2C4%2C9%2C19%2C20%2C23%2C24%2C38%2C42%27&COMMAND=%222020+OM4%3B%22&SOLAR_ELONG=%220%2C180%22&LHA_CUTOFF=12.0&CSV_FORMAT=YES&CAL_FORMAT=BOTH&ANG_FORMAT=DEG&APPARENT=AIRLESS&REF_SYSTEM=J2000&EXTRA_PREC=NO&CENTER=%27-1%27&START_TIME=%222020-10-19+00%3A00%22&STOP_TIME=%222020-11-19+00%3A00%22&STEP_SIZE=%221h%22&SKIP_DAYLT=NO (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f23fb5fa3c8>, 'Connection to ssd.jpl.nasa.gov timed out. (connect timeout=30)'))Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
MaxRetryError: HTTPSConnectionPool(host='ssd.jpl.nasa.gov', port=443): Max retries exceeded with url: /horizons_batch.cgi?batch=1&TABLE_TYPE=OBSERVER&QUANTITIES=%271%2C3%2C4%2C9%2C19%2C20%2C23%2C24%2C38%2C42%27&COMMAND=%222020+OM4%3B%22&SOLAR_ELONG=%220%2C180%22&LHA_CUTOFF=12.0&CSV_FORMAT=YES&CAL_FORMAT=BOTH&ANG_FORMAT=DEG&APPARENT=AIRLESS&REF_SYSTEM=J2000&EXTRA_PREC=NO&CENTER=%27-1%27&START_TIME=%222020-10-19+00%3A00%22&STOP_TIME=%222020-11-19+00%3A00%22&STEP_SIZE=%221h%22&SKIP_DAYLT=NO (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f23fb5fa3c8>, 'Connection to ssd.jpl.nasa.gov timed out. (connect timeout=30)'))Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f23fb5fa3c8>, 'Connection to ssd.jpl.nasa.gov timed out. (connect timeout=30)')Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
  File "/usr/local/lib64/python3.6/site-packages/gevent/_socket3.py", line 413, in connect
    self._wait(self._write_event)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
timeout: timed out
talister commented 3 years ago

@zemogle am I right in thinking we need to catch a whole bunch more exceptions from various places and maybe these will go away (or become less frequent). Looking at the stack traces (not sure what all the gevent stuff is; do we install/use that), it seems we need to do:

from requests import ConnectTimeout
from urllib3.exceptions import MaxRetryError, ConnectTimeoutError

in addition to the existing ConnectionError and then change the catch to:

    except (ConnectionError, ConnectTimout, MaxRetryError, ConnectTimeoutError) as e:
        logger.error("Unable to connect to HORIZONS")

This just seems... yuck...

zemogle commented 3 years ago

What a mess! I am surprised that the horizons library doesn't have a custom exception which would be much more useful.

We could use the explicit exceptions for all the system errors we know we might encounter, and then just catch the base classes. The errors classes you list should be subclasses of ConnectionError and TimeoutError.

except (ConnectionError, TimeoutError):
    logger.error('Horizons Timeout')
talister commented 2 years ago

Hopefully fixed by solution to Issue #586