ImagingDataCommons / ThrottleProxy

(CORE REPO)
Apache License 2.0
2 stars 0 forks source link

Connection reset by peer from REDIS #19

Closed wlongabaugh closed 4 years ago

wlongabaugh commented 4 years ago

Gettng 500 errors from the server. Per the issue at the bottom, Redis in Google cloud might be a bit flaky in App Engine Standard. Wrap the call to catch the exception. File "/srv/main.py", line 359, in root curr_use_per_ip_str = redis_client.get(client_ip) File "/env/lib/python3.7/site-packages/redis/client.py", line 1606, in get return self.execute_command('GET', name) File "/env/lib/python3.7/site-packages/redis/client.py", line 901, in execute_command return self.parse_response(conn, command_name, **options) File "/env/lib/python3.7/site-packages/redis/client.py", line 915, in parse_response response = connection.read_response() File "/env/lib/python3.7/site-packages/redis/connection.py", line 739, in read_response response = self._parser.read_response() File "/env/lib/python3.7/site-packages/redis/connection.py", line 324, in read_response raw = self._buffer.readline() File "/env/lib/python3.7/site-packages/redis/connection.py", line 256, in readline self._read_from_socket() File "/env/lib/python3.7/site-packages/redis/connection.py", line 223, in _read_from_socket (ex.args,)) redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer')"

See: https://github.com/andymccurdy/redis-py/issues/1186

wlongabaugh commented 4 years ago

Have submitted a support ticket to Google about this, though still need to harden the connection with retries.

wlongabaugh commented 4 years ago

Google reply was just to harden the connection attempt. Now tries up to 10 times, backing off with a progressively longer delay after a couple of immediate retries. Keep an eye open for this.