search

SolarArbiter / solarforecastarbiter-api

HTTP API and database schema for the Solar Forecast Arbiter
https://api.solarforecastarbiter.org
MIT License
10 stars 6 forks source link

Posting data sometimes fails #149

Closed sentry-io[bot] closed 4 years ago

sentry-io[bot] commented 5 years ago

with 500 error from the API. Seems that the connection to redis is timing out and better retries mechanisms should be in place.

Sentry Issue: CORE-18

August 28th 2019, 18:00:21.758sfa-api
[2019-08-29 01:00:21,757] ERROR in app: Exception on /observations/8c74582c-9a04-11e9-8ed6-0a580a820146/values [POST]
August 28th 2019, 18:00:21.758sfa-api
Traceback (most recent call last):
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
August 28th 2019, 18:00:21.758sfa-api
    response = self.full_dispatch_request()
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
August 28th 2019, 18:00:21.758sfa-api
    rv = self.handle_user_exception(e)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
August 28th 2019, 18:00:21.758sfa-api
    reraise(exc_type, exc_value, tb)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
August 28th 2019, 18:00:21.758sfa-api
    raise value
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
August 28th 2019, 18:00:21.758sfa-api
    rv = self.dispatch_request()
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
August 28th 2019, 18:00:21.758sfa-api
    return self.view_functions[rule.endpoint](**req.view_args)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/views.py", line 89, in view
August 28th 2019, 18:00:21.758sfa-api
    return self.dispatch_request(*args, **kwargs)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/flask/views.py", line 163, in dispatch_request
August 28th 2019, 18:00:21.758sfa-api
    return meth(*args, **kwargs)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/src/sfa_api/observations.py", line 264, in post
August 28th 2019, 18:00:21.758sfa-api
    base_url=request.url_root.rstrip('/'))
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/rq/queue.py", line 336, in enqueue
August 28th 2019, 18:00:21.758sfa-api
    at_front=at_front, meta=meta
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/rq/queue.py", line 286, in enqueue_call
August 28th 2019, 18:00:21.758sfa-api
    job = self.enqueue_job(job, at_front=at_front)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/rq/queue.py", line 362, in enqueue_job
August 28th 2019, 18:00:21.758sfa-api
    pipe.execute()
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/redis/client.py", line 3671, in execute
August 28th 2019, 18:00:21.758sfa-api
    self.shard_hint)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/redis/connection.py", line 1065, in get_connection
August 28th 2019, 18:00:21.758sfa-api
    if connection.can_read():
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/redis/connection.py", line 682, in can_read
August 28th 2019, 18:00:21.758sfa-api
    return self._parser.can_read(timeout)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/redis/connection.py", line 295, in can_read
August 28th 2019, 18:00:21.758sfa-api
    return self._buffer and self._buffer.can_read(timeout)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/redis/connection.py", line 205, in can_read
August 28th 2019, 18:00:21.758sfa-api
    raise_on_timeout=False)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/redis/connection.py", line 173, in _read_from_socket
August 28th 2019, 18:00:21.758sfa-api
    data = recv(self._sock, socket_read_size)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/redis/_compat.py", line 58, in recv
August 28th 2019, 18:00:21.758sfa-api
    return sock.recv(*args, **kwargs)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/gevent/_ssl3.py", line 491, in recv
August 28th 2019, 18:00:21.758sfa-api
    return self.read(buflen)
August 28th 2019, 18:00:21.758sfa-api
  File "/opt/app-root/lib/python3.7/site-packages/gevent/_ssl3.py", line 328, in read
August 28th 2019, 18:00:21.758sfa-api
    return self._sslobj.read(len or 1024)
August 28th 2019, 18:00:21.758sfa-api
ConnectionResetError: [Errno 104] Connection reset by peer
alorenzo175 commented 5 years ago

This is only happening on the dev-api for some reason. Is it an stunnel configuration issue?

alorenzo175 commented 4 years ago

stale and haven't seen in some time