search

goauthentik / authentik

The authentication glue you need.
https://goauthentik.io
Other
13.29k stars 887 forks source link

Using PGPool #6896

Closed boesr closed 1 year ago

boesr commented 1 year ago

Describe your question/ I am running Pgpool II before my database. It was working with version 2023.6.2, but now fails. When running with AUTHENTIK_POSTGRESQL__USE_PGBOUNCER=true the frontend loads all data once, but just hangs afterwards. PGpool shows an error, which indicates, that the frontend (Authentik) cannot use authentication methods offered by pgpool (see https://www.pgpool.net/pipermail/pgpool-general/2022-June/008262.html). Without that parameter I just get CSS written on the login page and the frontend isn't usable at all. The logs show the error below.

Do I have to switch to certain authentication methods or what did change from 2023.6.2 to 2023.8.x regarding that fact?

Relevant infos 2023.8.3

Screenshots /

Logs

Error when not using PGBOUNCER parameter:

{"event": "Internal Server Error: /api/v3/outposts/instances/", "exception": [{"exc_type": "OperationalError", "exc_value": "consuming input failed: unable to execute Describe\nDETAIL:  unable to get the bind message\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.", "frames": [{"filename": "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", "line": "", "lineno": 534, "locals": {"__traceback_hide__": "True", "args": "'(functools.partial(<function response_for_exception at 0x7f941bda8fe0>, <ASGIReq'+298", "current_thread": "<Thread(ThreadPoolExecutor-149_0, started 140273531205312)>", "exc_info": "\"(<class 'django.db.utils.OperationalError'>, OperationalError('consuming input f\"+257", "func": "<built-in method run of _contextvars.Context object at 0x7f94102ff180>", "kwargs": "{}", "loop": "<uvloop.Loop running=True closed=False debug=False>", "parent_set": "True", "self": "<asgiref.sync.SyncToAsync object at 0x7f941046d2d0>", "source_task": "None"}, "name": "thread_handler"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", "line": "", "lineno": 42, "locals": {"exc": "\"OperationalError('consuming input failed: unable to execute Describe\\\\nDETAIL:  u\"+173", "get_response": "'<bound method BaseHandler._get_response_async of <django.core.handlers.asgi.ASGI'+34", "request": "<ASGIRequest: GET '/api/v3/outposts/instances/'>"}, "name": "inner"}, {"filename": "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", "line": "", "lineno": 534, "locals": {"__traceback_hide__": "True", "args": "'(functools.partial(<bound method BaseHandler.process_exception_by_middleware of '+373", "current_thread": "<Thread(ThreadPoolExecutor-147_0, started 140273799640768)>", "exc_info": "\"(<class 'django.db.utils.OperationalError'>, OperationalError('consuming input f\"+257", "func": "<built-in method run of _contextvars.Context object at 0x7f94103149c0>", "kwargs": "{}", "loop": "<uvloop.Loop running=True closed=False debug=False>", "parent_set": "False", "self": "<asgiref.sync.SyncToAsync object at 0x7f941046e410>", "source_task": "None"}, "name": "thread_handler"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", "line": "", "lineno": 253, "locals": {"callback": "<function OutpostViewSet at 0x7f94110b7560>", "callback_args": "()", "callback_kwargs": "{}", "middleware_method": "<asgiref.sync.SyncToAsync object at 0x7f9410f7c490>", "request": "<ASGIRequest: GET '/api/v3/outposts/instances/'>", "response": "None", "self": "<django.core.handlers.asgi.ASGIHandler object at 0x7f9411473c90>", "wrapped_callback": "<asgiref.sync.SyncToAsync object at 0x7f94103418d0>"}, "name": "_get_response_async"}, {"filename": "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", "line": "", "lineno": 479, "locals": {"__traceback_hide__": "True", "args": "(<ASGIRequest: GET '/api/v3/outposts/instances/'>,)", "child": "'functools.partial(<function OutpostViewSet at 0x7f94110b7560>, <ASGIRequest: GET'+32", "context": "<_contextvars.Context object at 0x7f941041fec0>", "executor": "<asgiref.current_thread_executor.CurrentThreadExecutor object at 0x7f941034ea10>", "func": "<built-in method run of _contextvars.Context object at 0x7f941041fec0>", "kwargs": "{}", "loop": "<uvloop.Loop running=True closed=False debug=False>", "self": "<asgiref.sync.SyncToAsync object at 0x7f94103418d0>"}, "name": "__call__"}, {"filename": "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", "line": "", "lineno": 40, "locals": {"__traceback_hide__": "True", "self": "None"}, "name": "run"}, {"filename": "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", "line": "", "lineno": 538, "locals": {"__traceback_hide__": "True", "args": "'(functools.partial(<function OutpostViewSet at 0x7f94110b7560>, <ASGIRequest: GE'+35", "current_thread": "<Thread(ThreadPoolExecutor-147_0, started 140273799640768)>", "exc_info": "(None, None, None)", "func": "<built-in method run of _contextvars.Context object at 0x7f941041fec0>", "kwargs": "{}", "loop": "<uvloop.Loop running=True closed=False debug=False>", "parent_set": "False", "self": "<asgiref.sync.SyncToAsync object at 0x7f94103418d0>", "source_task": "None"}, "name": "thread_handler"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", "line": "", "lineno": 56, "locals": {"args": "(<ASGIRequest: GET '/api/v3/outposts/instances/'>,)", "kwargs": "{}", "view_func": "<function OutpostViewSet at 0x7f94110b74c0>"}, "name": "wrapper_view"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework/viewsets.py", "line": "", "lineno": 125, "locals": {"action": "list", "actions": "{'get': 'list', 'post': 'create', 'head': 'list'}", "args": "()", "cls": "<class 'authentik.outposts.api.outposts.OutpostViewSet'>", "handler": "'<bound method ListModelMixin.list of <authentik.outposts.api.outposts.OutpostVie'+31", "initkwargs": "{'suffix': 'List', 'basename': 'outpost', 'detail': False}", "kwargs": "{}", "method": "head", "request": "<ASGIRequest: GET '/api/v3/outposts/instances/'>", "self": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "view"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", "line": "", "lineno": 509, "locals": {"args": "()", "handler": "'<bound method ListModelMixin.list of <authentik.outposts.api.outposts.OutpostVie'+31", "kwargs": "{}", "request": "<rest_framework.request.Request: GET '/api/v3/outposts/instances/'>", "self": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "dispatch"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", "line": "", "lineno": 469, "locals": {"context": "\"{'view': <authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff5\"+109", "exc": "\"OperationalError('consuming input failed: unable to execute Describe\\\\nDETAIL:  u\"+173", "exception_handler": "<function exception_handler at 0x7f9414529a80>", "response": "None", "self": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "handle_exception"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", "line": "", "lineno": 480, "locals": {"exc": "\"OperationalError('consuming input failed: unable to execute Describe\\\\nDETAIL:  u\"+173", "self": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "raise_uncaught_exception"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", "line": "", "lineno": 506, "locals": {"args": "()", "handler": "'<bound method ListModelMixin.list of <authentik.outposts.api.outposts.OutpostVie'+31", "kwargs": "{}", "request": "<rest_framework.request.Request: GET '/api/v3/outposts/instances/'>", "self": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "dispatch"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", "line": "", "lineno": 38, "locals": {"args": "()", "kwargs": "{}", "request": "<rest_framework.request.Request: GET '/api/v3/outposts/instances/'>", "self": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "list"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework/generics.py", "line": "", "lineno": 150, "locals": {"backend": "<class 'rest_framework_guardian.filters.ObjectPermissionsFilter'>", "queryset": "<QuerySet [<Outpost: Outpost authentik Embedded Outpost>]>", "self": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "filter_queryset"}, {"filename": "/usr/local/lib/python3.11/site-packages/rest_framework_guardian/filters.py", "line": "", "lineno": 28, "locals": {"get_objects_for_user": "<function get_objects_for_user at 0x7f9419b15e40>", "permission": "authentik_outposts.view_outpost", "queryset": "<QuerySet [<Outpost: Outpost authentik Embedded Outpost>]>", "request": "<rest_framework.request.Request: GET '/api/v3/outposts/instances/'>", "self": "'<rest_framework_guardian.filters.ObjectPermissionsFilter object at 0x7f94102f539'+2", "user": "<User: ak-outpost-81160a44205649d0a8ddcab0fc7ba648>", "view": "<authentik.outposts.api.outposts.OutpostViewSet object at 0x7f941034ff50>"}, "name": "filter_queryset"}, {"filename": "/usr/local/lib/python3.11/site-packages/guardian/shortcuts.py", "line": "", "lineno": 542, "locals": {"accept_global_perms": "False", "any_perm": "False", "app_label": "authentik_outposts", "codename": "view_outpost", "codenames": "{'view_outpost'}", "ctype": "<ContentType: authentik_outposts | outpost>", "klass": "<QuerySet [<Outpost: Outpost authentik Embedded Outpost>]>", "new_app_label": "authentik_outposts", "new_ctype": "<ContentType: authentik_outposts | outpost>", "perm": "authentik_outposts.view_outpost", "perms": "['authentik_outposts.view_outpost']", "queryset": "<QuerySet [<Outpost: Outpost authentik Embedded Outpost>]>", "use_groups": "True", "user": "<User: ak-outpost-81160a44205649d0a8ddcab0fc7ba648>", "with_superuser": "True"}, "name": "get_objects_for_user"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", "line": "", "lineno": 57, "locals": {"cls": "<class 'authentik.core.models.User'>", "instance": "<User: ak-outpost-81160a44205649d0a8ddcab0fc7ba648>", "self": "<django.utils.functional.cached_property object at 0x7f9419532f10>"}, "name": "__get__"}, {"filename": "/authentik/core/models.py", "line": "", "lineno": 211, "locals": {"self": "<User: ak-outpost-81160a44205649d0a8ddcab0fc7ba648>"}, "name": "is_superuser"}, {"filename": "/authentik/core/models.py", "line": "", "lineno": 163, "locals": {"self": "<User: ak-outpost-81160a44205649d0a8ddcab0fc7ba648>"}, "name": "all_groups"}, {"filename": "/authentik/core/models.py", "line": "", "lineno": 163, "locals": {".0": "<generator object QuerySet._iterator at 0x7f9410e9e460>"}, "name": "<genexpr>"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", "line": "", "lineno": 516, "locals": {"chunk_size": "None", "iterable": "<django.db.models.query.FlatValuesListIterable object at 0x7f94102f5490>", "self": "<QuerySet []>", "use_chunked_fetch": "True"}, "name": "_iterator"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", "line": "", "lineno": 285, "locals": {"compiler": "\"<SQLCompiler model=Group connection=<DatabaseWrapper vendor='postgresql' alias='\"+26", "queryset": "<QuerySet []>", "self": "<django.db.models.query.FlatValuesListIterable object at 0x7f94102f5490>"}, "name": "__iter__"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", "line": "", "lineno": 1513, "locals": {"chunk_size": "2000", "chunked_fetch": "True", "results": "None", "self": "\"<SQLCompiler model=Group connection=<DatabaseWrapper vendor='postgresql' alias='\"+26", "tuple_expected": "False"}, "name": "results_iter"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", "line": "", "lineno": 1562, "locals": {"chunk_size": "2000", "chunked_fetch": "True", "cursor": "<django.db.backends.utils.CursorWrapper object at 0x7f9410346290>", "params": "(7,)", "result_type": "multi", "self": "\"<SQLCompiler model=Group connection=<DatabaseWrapper vendor='postgresql' alias='\"+26", "sql": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172"}, "name": "execute_sql"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", "line": "", "lineno": 67, "locals": {"params": "(7,)", "self": "<django.db.backends.utils.CursorWrapper object at 0x7f9410346290>", "sql": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172"}, "name": "execute"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", "line": "", "lineno": 80, "locals": {"context": "\"{'connection': <DatabaseWrapper vendor='postgresql' alias='default'>, 'cursor': \"+66", "executor": "'<bound method CursorWrapper._execute of <django.db.backends.utils.CursorWrapper '+26", "many": "False", "params": "(7,)", "self": "<django.db.backends.utils.CursorWrapper object at 0x7f9410346290>", "sql": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172"}, "name": "_execute_with_wrappers"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", "line": "", "lineno": 84, "locals": {"ignored_wrapper_args": "\"(False, {'connection': <DatabaseWrapper vendor='postgresql' alias='default'>, 'c\"+75", "params": "(7,)", "self": "<django.db.backends.utils.CursorWrapper object at 0x7f9410346290>", "sql": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172"}, "name": "_execute"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/utils.py", "line": "", "lineno": 91, "locals": {"db_exc_type": "<class 'psycopg.OperationalError'>", "dj_exc_type": "<class 'django.db.utils.OperationalError'>", "dj_exc_value": "\"OperationalError('consuming input failed: unable to execute Describe\\\\nDETAIL:  u\"+173", "exc_type": "<class 'psycopg.OperationalError'>", "exc_value": "\"OperationalError('consuming input failed: unable to execute Describe\\\\nDETAIL:  u\"+173", "self": "<django.db.utils.DatabaseErrorWrapper object at 0x7f9410340810>", "traceback": "<traceback object at 0x7f941044db80>"}, "name": "__exit__"}, {"filename": "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", "line": "", "lineno": 89, "locals": {"ignored_wrapper_args": "\"(False, {'connection': <DatabaseWrapper vendor='postgresql' alias='default'>, 'c\"+75", "params": "(7,)", "self": "<django.db.backends.utils.CursorWrapper object at 0x7f9410346290>", "sql": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172"}, "name": "_execute"}, {"filename": "/usr/local/lib/python3.11/site-packages/psycopg/server_cursor.py", "line": "", "lineno": 294, "locals": {"binary": "None", "kwargs": "{}", "params": "(7,)", "query": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172", "self": "\"<psycopg.ServerCursor '_django_curs_140273799640768_sync_1' [closed] [BAD] at 0x\"+13"}, "name": "execute"}], "is_cause": false, "syntax_error": null}, {"exc_type": "OperationalError", "exc_value": "consuming input failed: unable to execute Describe\nDETAIL:  unable to get the bind message\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.", "frames": [{"filename": "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", "line": "", "lineno": 89, "locals": {"ignored_wrapper_args": "\"(False, {'connection': <DatabaseWrapper vendor='postgresql' alias='default'>, 'c\"+75", "params": "(7,)", "self": "<django.db.backends.utils.CursorWrapper object at 0x7f9410346290>", "sql": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172"}, "name": "_execute"}, {"filename": "/usr/local/lib/python3.11/site-packages/psycopg/server_cursor.py", "line": "", "lineno": 294, "locals": {"binary": "None", "kwargs": "{}", "params": "(7,)", "query": "'SELECT \"authentik_core_group\".\"group_uuid\" FROM \"authentik_core_group\" INNER JOI'+172", "self": "\"<psycopg.ServerCursor '_django_curs_140273799640768_sync_1' [closed] [BAD] at 0x\"+13"}, "name": "execute"}], "is_cause": true, "syntax_error": null}], "level": "error", "logger": "django.request", "timestamp": 1694687834.2798047}

With the parameter pgpool shows the following in the log: 

2023-09-14 14:40:01.828: [unknown] pid 38911: ERROR:  unable to read data from frontend
2023-09-14 14:40:01.829: [unknown] pid 38911: DETAIL:  EOF encountered with frontend

Version and Deployment (please complete the following information):

Additional context /

BeryJu commented 1 year ago

The only thing I could imagine causing issues with this is https://github.com/goauthentik/authentik/commit/95458570425223a82285bb3ea04ae233aedba54d, but that was reverted with 2023.8.2+

boesr commented 1 year ago

Thanks for the hint. I am going to investigate that further. At the moment I could see that when increasing the connection limit the pgbouncer option does work. At least till I logout and login again (sometimes twice). This leads to exceeding the postgresql connection limit. I try to increase that to have a walkaround till I find another solution.

I also tried so switch back to 2023.6.2. Then everything works again.

boesr commented 1 year ago

Couldn't fix the issue without setting the AUTHENTIK_POSTGRESQL__USE_PGBOUNCER=true parameter.

I could increase the connection limit and the shared_buffers value to create a walkaround for the problem. Not quiet sure if it even is a walkaround or if it is necessary.

For others facing the same problem:

Seems like authentik in pgbouncer mode is able to open more connections than pgpool could handle in default, which leads to a freezing ui. I couldn't reproduce the error with that configuration.

Still no clue why the mode without pgbouncer isn't working anymore.

I think that could be closed for now since there is a way to fix it.

boesr commented 1 year ago

Seems like the walkaround does not work. It just delays the problem. It seems when using AUTHENTIK_POSTGRESQL__USE_PGBOUNCER=true some connections won't be closed while new ones are opened leading to a freezing db / ui.

A restart of Authentik fixes that freeze.

@BeryJu I would keep that issue open and investigate that further. Maybe I can find a solution. I will update the issue if so.

boesr commented 1 year ago

solved in https://github.com/goauthentik/authentik/pull/6949