django.db.utils.OperationalError: SSL SYSCALL error: EOF detected

elpaso commented 3 years ago

Hi,

I'm testing this module with --worker-type process and 5 workers.

Huey consumer started with 5 process, PID 338935 at 2021-04-01 09:21:11.148253
Scheduler runs every 1 second(s).
Periodic tasks are enabled.
The following commands are available:
+ openrouteservice.tasks.isochrone_from_layer_task
Checking periodic tasks
Sleeping for 0.9916734490107046
calling startup hook "startup_handler"
calling startup hook "startup_handler"
calling startup hook "startup_handler"
calling startup hook "startup_handler"
calling startup hook "startup_handler"
startup hook "startup_handler" failed
Traceback (most recent call last):
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: SSL error: decryption failed or bad record mac

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ale/.local/lib/python3.8/site-packages/huey/consumer.py", line 94, in initialize
    startup_hook()
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/huey_monitor/tasks.py", line 78, in startup_handler
    for task_model_instance in qs:
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
    cursor.execute(sql, params)
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
    return super().execute(sql, params)
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: SSL error: decryption failed or bad record mac

startup hook "startup_handler" failed
Traceback (most recent call last):
  File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

Note that the error does not appear when I use a single worker.

elpaso commented 3 years ago

The check for Worker-1 fixes the startup issue for me but I suspect that there is more: shouldn't you close the connection as the db_task decorator does?


@on_startup()
def startup_handler():

    if multiprocessing.current_process().name == 'Worker-1':

        logger.debug('startup handler called for %s' %
                     multiprocessing.current_process().name)

        with transaction.atomic():

jedie commented 3 years ago

Hm... SSL error sounds like a other not related error than not closed connections... Do you have use out test docker-compose project via make up ?

I changed https://github.com/boxine/django-huey-monitor/blob/41fa9f163bb11caf5da7fa50d7490f50daaa93c0/docker/huey/entrypoint.sh#L29 and also set --workers 5 ... I add a print(len(connections.all())) in both existing signal handlers... It prints always 1 ... So i think a explicit handling is not needed here, isn't it?!?

boxine / django-huey-monitor

django.db.utils.OperationalError: SSL SYSCALL error: EOF detected #24