wgentine
commented
5 months ago Having same issue after migrating from 2024.4 to 2024.4.1
Closed FutureFlySpace closed 2 months ago
wgentine
commented
5 months ago Having same issue after migrating from 2024.4 to 2024.4.1
wgentine
commented
5 months ago Entering an infinite login loop:
INF | action=system_exception auth_via=unauthenticated client_ip=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx context={"asn":{"as_org":"KPN B.V.","asn":1136,"network":"2a02:a456::/32"},"geo":{"city":"","continent":"EU","country":"NL","lat":52.,"long":4.},"http_request":{"args":{},"method":"GET","path":"/api/v3/core/users/10/","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"},"message":"Traceback (most recent call last):\n File \"/ak-root/venv/lib/python3.12/site-packages/asgiref/sync.py\", line 534, in thread_handler\n raise exc_info[1]\n File \"/ak-root/venv/lib/python3.12/site-packages/django/core/handlers/base.py\", line 253, in _get_response_async\n response = await wrapped_callback(\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/asgiref/sync.py\", line 479, in call\n ret: _R = await loop.run_in_executor(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/asgiref/current_thread_executor.py\", line 40, in run\n result = self.fn(*self.args, self.kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/asgiref/sync.py\", line 538, in thread_handler\n return func(*args, *kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/integrations/django/views.py\", line 84, in sentry_wrapped_callback\n return callback(request, args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/django/views/decorators/csrf.py\", line 65, in _view_wrapper\n return view_func(request, *args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/viewsets.py\", line 125, in view\n return self.dispatch(request, *args, *kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/views.py\", line 509, in dispatch\n response = self.handle_exception(exc)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/views.py\", line 469, in handle_exception\n self.raise_uncaught_exception(exc)\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/views.py\", line 480, in raise_uncaught_exception\n raise exc\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/views.py\", line 506, in dispatch\n response = handler(request, args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/mixins.py\", line 54, in retrieve\n instance = self.get_object()\n ^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/generics.py\", line 83, in get_object\n queryset = self.filter_queryset(self.get_queryset())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/ak-root/venv/lib/python3.12/site-packages/rest_framework/generics.py\", line 150, in filter_queryset\n queryset = backend().filter_queryset(self.request, queryset, self)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/authentik/rbac/filters.py\", line 28, in filter_queryset\n if request.user.type == UserTypes.INTERNAL_SERVICE_ACCOUNT:\n ^^^^^^^^^^^^^^^^^\nbuiltins.AttributeError: 'AnonymousUser' object has no attribute 'type'"} domain_url=auth.xxxxx.nl event=Created Event host=auth.xxxx.nl logger=authentik.events.models pid=46 request_id=c47bb557fd4b4b8880f330ed6e1acf8a schema_name=public timestamp=2024-04-27T07:00:00.192737 user={"email":"","is_anonymous":true,"pk":1,"username":"AnonymousUser"}
wgentine
commented
5 months ago And this:
warning | domain_url=null event=Task failure exc=OperationalError('deadlock detected\nDETAIL: Process 3744 waits for ShareLock on transaction 8427503; blocked by process 3741.\nProcess 3741 waits for ShareLock on transaction 8427521; blocked by process 3744.\nHINT: See server log for query details.\nCONTEXT: while locking tuple (5,1) in relation "authentik_stages_authenticator_webauthn_webauthndevicetype"') logger=authentik.root.celery pid=47 schema_name=public task_id=task-72d94e8eaaa04f839a9e7f0d6d539c61 timestamp=2024-04-27T07:18:42.368168
ERR | event=Task authentik.stages.authenticator_webauthn.tasks.webauthn_aaguid_import[72d94e8e-aaa0-4f83-9a9e-7f0d6d539c61] raised unexpected: OperationalError('deadlock detected\nDETAIL: Process 3744 waits for ShareLock on transaction 8427503; blocked by process 3741.\nProcess 3741 waits for ShareLock on transaction 8427521; blocked by process 3744.\nHINT: See server log for query details.\nCONTEXT: while locking tuple (5,1) in relation "authentik_stages_authenticator_webauthn_webauthndevicetype"') exception=[{"exc_type":"OperationalError","exc_value":"deadlock detected\nDETAIL: Process 3744 waits for ShareLock on transaction 8427503; blocked by process 3741.\nProcess 3741 waits for ShareLock on transaction 8427521; blocked by process 3744.\nHINT: See server log for query details.\nCONTEXT: while locking tuple (5,1) in relation \"authentik_stages_authenticator_webauthn_webauthndevicetype\"","frames":[{"filename":"/ak-root/venv/lib/python3.12/site-packages/celery/app/trace.py","line":"","lineno":453,"locals":{},"name":"trace_task"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/integrations/celery.py","line":"","lineno":325,"locals":{},"name":"_inner"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/_compat.py","line":"","lineno":127,"locals":{},"name":"reraise"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/integrations/celery.py","line":"","lineno":320,"locals":{},"name":"_inner"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/celery/app/trace.py","line":"","lineno":736,"locals":{},"name":"__protected_call"},{"filename":"/authentik/stages/authenticator_webauthn/tasks.py","line":"","lineno":66,"locals":{},"name":"webauthn_aaguid_import"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/manager.py","line":"","lineno":87,"locals":{},"name":"manager_method"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/query.py","line":"","lineno":986,"locals":{},"name":"update_or_create"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/query.py","line":"","lineno":948,"locals":{},"name":"get_or_create"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/query.py","line":"","lineno":645,"locals":{},"name":"get"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/query.py","line":"","lineno":382,"locals":{},"name":"len"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/query.py","line":"","lineno":1928,"locals":{},"name":"_fetch_all"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/query.py","line":"","lineno":91,"locals":{},"name":"iter"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py","line":"","lineno":1562,"locals":{},"name":"execute_sql"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/integrations/django/init.py","line":"","lineno":644,"locals":{},"name":"execute"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py","line":"","lineno":79,"locals":{},"name":"execute"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py","line":"","lineno":92,"locals":{},"name":"_execute_with_wrappers"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py","line":"","lineno":100,"locals":{},"name":"_execute"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/utils.py","line":"","lineno":91,"locals":{},"name":"exit"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py","line":"","lineno":105,"locals":{},"name":"_execute"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py","line":"","lineno":69,"locals":{},"name":"execute"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py","line":"","lineno":732,"locals":{},"name":"execute"}],"is_cause":false,"syntax_error":null},{"exc_type":"DeadlockDetected","exc_value":"deadlock detected\nDETAIL: Process 3744 waits for ShareLock on transaction 8427503; blocked by process 3741.\nProcess 3741 waits for ShareLock on transaction 8427521; blocked by process 3744.\nHINT: See server log for query details.\nCONTEXT: while locking tuple (5,1) in relation \"authentik_stages_authenticator_webauthn_webauthndevicetype\"","frames":[{"filename":"/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py","line":"","lineno":105,"locals":{"ignored_wrapper_args":"\"(False, {'connection':
AurelienGoor
commented
5 months ago Samething The issue appeared after the migration from 2024.2.3 to 2024.4.0. I updated to 2024.4.1 but the issue is still present.
Repro steps:
if/flow/default-authentication-flowor sometimes when I go to the Admin dashboard, I am automatically redirected to the login page.
I point out that Authentik is deployed in a Docker container and it is behing a reverse proxy (Nginx proxy manager).
Log:
INF auth_via=unauthenticated domain_url=0.0.0.0 event=/-/health/live/ host=0.0.0.0:9000 logger=authentik.asgi method=HEAD pid=37434 remote=127.0.0.1 request_id=178489a5ede94fbf920375565767e49a runtime=22 schema_name=public scheme=http status=204 timestamp=2024-04-27T16:00:59.208815 user= user_agent=goauthentik.io/healthcheck
INF domain_url=null event=/ws/client/ logger=authentik.asgi pid=37434 remote=XXX.XXX.XXX.XXX schema_name=public scheme=ws timestamp=2024-04-27T16:01:21.279727 user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
wgentine
commented
5 months ago @AurelienGoor I guess there might be way more people with this issue then! exactly same problem. Decided to deploy another environment with the 2024.4.1 version and it works. So the upgrade messed up with something and I spent hours but couldnt find the root cause yet.
wgentine
commented
5 months ago OK, I managed to solve this issue.
I saw some other issues that were reported the table authentik_install_id had 2 ids and checked that mine had too. Checked which value was reported through the API: /api/v3/enterprise/license/get_install_id/ and deleted the other one.
Restarted everything and got it working again.
wgentine
commented
5 months ago Celebrated too early. Its definitely better since I can keep logged in for a little more time. Also upgraded to beta branch... will wait for the devs to say something. This is really messed up.
wgentine
commented
5 months ago Dont ask me why but workaround: stop redis container.
FutureFlySpace
commented
5 months ago I can confirm this. After stopping the Redis container, I was able to access authentik again.
Thanks for your effort @wgentine Hopefully there will be a reasonable fix soon 😃
wgentine
commented
5 months ago @FutureFlySpace I hope so... Now it intrigues me what's going on... and why is redis needed anyways because I see no loss of performance but redis must be running for the other containers to start, then you kill it.
AurelienGoor
commented
5 months ago Thanks for the workaround.
I tried to delete the redis database and to restart the redis and other containers, but the issue occurs again...
wgentine
commented
5 months ago Don't restart the container. Start everything then stop Redis and let it stopped.
wgentine
commented
5 months ago @BeryJu are you aware of what could be the cause here? Thanks!
wgentine
commented
5 months ago In the meanwhile, even better workaround: set the parameter AUTHENTIK_SESSION_STORAGE=db Then you dont have to stop redis.
wgentine
commented
5 months ago Zero attention for this issue. Pity.
wgentine
commented
5 months ago Has anyone found a fix?
wgentine
commented
4 months ago Still broken with 2024.4.2
FutureFlySpace
commented
4 months ago I don't know how or why, but it's working for me again now. I haven't had the stack running for about 2 weeks. Started and upgraded it today. So far I have no more problems. Very strange
FutureFlySpace
commented
4 months ago Well nevermind, now it's not working again and I'm stuck in a login loop. No errors in the browser console. A restart helped.
jribal
commented
4 months ago Same issue here.
AUTHENTIK_SESSION_STORAGE=db works for me but my dashboard shows:
No workers connected. Background tasks will not run.
Forget password are not send.
Commenting AUTHENTIK_SESSION_STORAGE then stopping the redis does not show the error and allows to send forget password emails.
Edit:
My installation: docker-compose + traefik
I can sometimes connect with redis configured but 3 refreshes with CTRL+R always sends me to the login page.
I started redis with -loglevel verbose and I can see redis activity, no error anywhere.
I made some tests with AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS and traefik trustedIps, no effect.
Edit2:
It might related to redis offloading data to file, so far changing the command from --save 300 to --save "" works much longer. I will report again when it fails.
FutureFlySpace
commented
4 months ago After a few weeks of Authentik running, it crashed today. When I restarted the stack, I got the following message from Redis:
authentik-redis | 1:C 04 Jun 2024 07:23:48.965 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.Perhaps someone can say something about this?
kenneth-ellis
commented
3 months ago I was experiencing symptoms (constant reset / never-ending login loops, failed-to-fetch resources on the rare occasion I managed to get past the login screen, etc) similar to those described here on a completely clean fresh install of Authentik 2024.4.2. Issues persisted despite shutting down and recreating containers, rebooting the entire docker server. At some point during troubleshooting I shut down the only other app I had running in docker at the time -- Immich. All of the issues with Authentik being unstable/unusable immediately went away. Immich also uses Postgres and Redis with the same tcp ports but it has unique container names so they shouldn't be conflicting with each other. Somehow in my case they seem to be, though. I haven't delved into troubleshooting too much more deeply yet but I'm curious if anyone else here is also running Immich alongside Authentik and if shutting Immich down temporarily resolves your issues as well.
jribal
commented
3 months ago @kenneth-ellis That way seems to work for me. I'm not sure why changing the command was working for nearly a month. Just rename all the redis name to authentik-redis works for me. I'm pretty sure I tried this without success before.
@FutureFlySpace All my redis instances have this log. My other servers do not have a problem.
FutureFlySpace
commented
2 months ago Since nothing seems to be happening in this issue anyway and the problem has somehow solved itself for me, I will close the issue. Thanks anyway for the help
Describe the bug When I call up Authentik, I get to
/if/flow/default-authentication-flow/. Here I log in normally. I am then redirected to the dashboard, but nothing loads. The browser console shows that the API cannot be called because I have no access authorizations (forbidden). Also, the title of the page when I log in is nowWelcome to authentik!. I had changed this before the upgradeTo Reproduce Steps to reproduce the behavior:
Expected behavior Dashboard loads and I am authorized to access the API
Screenshots
Logs
No errors or anything, just this the whole time:
Version and Deployment:
Additional context
The browser console log