🐛 Garbage collector does not handle PG unavailability gracefully.

[mrnicegyu11]

Garbage collector does not handle PG unavailability gracefully.

Incident report:

• Today at approx 9:45 (Zurich Time) Taylor was forcefully logged out of his study (UUID 6e4ea1a4-b02c-11ec-b9d7-02420a0b0063) on dalco-staging. These errors where discovered in the following investigation.

• During this time there were multiple network errors and package drops in Z43. These are the main cause of the errors.

• There were many aiopg timeouts.

• This is the stack-trace of one exception thrown by the garbage colelctor running remove_users_manually_marked_as_guests, which ERRORED with a stack trace as such:

  "File ""/home/scu/.venv/lib/python3.9/site-packages/simcore_service_webserver/garbage_collector_core.py"", line 78, in collect_garbage",
  "guest_users: list[tuple[int, str]] = await get_guest_user_ids_and_names(app)",
  "[1;33mWARNING: [2022-07-20 07:40:41,153/MainProcess] [simcore_service_webserver.garbage_collector_task:collect_garbage_periodically(89)]  -  Stopped: There was an error during garbage collection, restarting...",
  "return fut.result()",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/simcore_service_webserver/garbage_collector_task.py"", line 73, in collect_garbage_periodically",
  "The above exception was the direct cause of the following exception:",
  "asyncio.exceptions.CancelledError",
  "await collect_garbage(app)",
  "Traceback (most recent call last):",
  "File ""/usr/local/lib/python3.9/asyncio/tasks.py"", line 490, in wait_for",
  "Traceback (most recent call last):",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/simcore_service_webserver/garbage_collector_core.py"", line 256, in remove_users_manually_marked_as_guests",
  "await remove_users_manually_marked_as_guests(registry, app)",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/aiopg/connection.py"", line 426, in execute",
  "await cursor.execute(str(compiled), post_processed_params[0])",
  "raise exc",
  "File ""/usr/local/lib/python3.9/asyncio/tasks.py"", line 492, in wait_for",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/aiopg/connection.py"", line 881, in _poll",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/aiopg/connection.py"", line 884, in _poll",
  "await self._conn._poll(waiter, timeout)",
  "asyncio.exceptions.TimeoutError",
  "await asyncio.wait_for(self._waiter, timeout)",
  "raise exceptions.TimeoutError() from exc",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/aiopg/sa/connection.py"", line 168, in _execute",
  "self._obj = await self._coro",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/simcore_service_webserver/users_api.py"", line 157, in get_guest_user_ids_and_names",
  "async for row in conn.execute(",
  "File ""/home/scu/.venv/lib/python3.9/site-packages/aiopg/utils.py"", line 115, in __anext__",

Actionable follow-up: PG being unreachable needs to be handled gracefully throughout the app:

• [ ] garbage collector retries if pg connections time out
• [ ] after a suitable number of retrails, the GC logs a unique and descriptive error, for which an alert in graylog can be devised

[pcrespov]

@sanderegg do you mind following up this one in my absence? I will start having a look and leave some notes here of my findings.

[GitHK]

@matusdrobuliak66 @pcrespov is this still a thing or did we add something to check connections to PG?