search

jwbensley / dfz_name_and_shame

DFZ Name and Shame - A Twitter bot which tweets stats about the BGP DFZ.
3 stars 0 forks source link

Not enough memory for Redis #193

Closed jwbensley closed 10 months ago

jwbensley commented 1 year ago

DNAS is crashing because Redis is unreachable, Redis is unreachable because the VM is running out of memory:

    json_str = self.r.get(key)
               ^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/commands/core.py", line 1790, in get
    return self.execute_command("GET", name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/client.py", line 1258, in execute_command
    return conn.retry.call_with_retry(
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/retry.py", line 49, in call_with_retry
    fail(error)
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/client.py", line 1262, in <lambda>
    lambda error: self._disconnect_raise(conn, error),
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/client.py", line 1248, in _disconnect_raise
    raise error
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
           ^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/client.py", line 1259, in <lambda>
    lambda: self._send_command_parse_response(
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/client.py", line 1235, in _send_command_parse_response
    return self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/client.py", line 1275, in parse_response
    response = connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/connection.py", line 848, in read_response
    response = self._parser.read_response(disable_decoding=disable_decoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/connection.py", line 335, in read_response
    result = self._read_response(disable_decoding=disable_decoding)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/connection.py", line 377, in _read_response
    response = self._buffer.read(length)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/connection.py", line 230, in read
    self._read_from_socket(length - self.length)
  File "/opt/pypy3.10-v7.3.12-linux64/lib/pypy3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
ERROR: 1
jwbensley commented 12 months ago

This is crashing because the REDIS backups cause OOM killer to kick in.

jwbensley commented 10 months ago

Fixed by compressing the data in redis.