goauthentik / authentik

The authentication glue you need.
https://goauthentik.io
Other
13.67k stars 915 forks source link

Authentik and Redis disconnect 30 seconds after Authentik is brought online. #12107

Open Aetherinox opened 1 day ago

Aetherinox commented 1 day ago

Describe the bug As of recent, Authentik keeps dropping the connection to Redis. Attemping to use docker container name for Redis which is rs01. Within 30 or so seconds of bringing Authentik up, an error about Redis is immediately thrown, and then within another 30 seconds, the health status of authentik-worker goes to Unhealthy.

Have used the same setup for a year now. Container names haven't changed, Redis password is the same, nothing in the docker-compose.yml for either image has changed.

ERR | event=consumer: Cannot connect to redis://:**@rs01:6379/0: Error 111 connecting to rs01:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)

Restarting the container presents the same results.

Have connected to the same Redis container for testing with:

All other applications plugged into that Redis container work fine, except for Authentik.

To Reproduce Steps to reproduce the behavior:

  1. Set up Authentik with integrated outpost. Consists of two containers:

    authentik-server:
        container_name: ${AUTHENTIK_CONTAINER_NAME:-authentik-server}
        image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.10.2}
        command: server
    
    authentik-worker:
        container_name: ${AUTHENTIK_WORKER_CONTAINER_NAME:-authentik-worker}
        image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.10.2}
        command: worker
  2. Bring up Redis container

  3. Bring up Authentik container

  4. Error appears after 30 seconds - 1 minute

Expected behavior Authentik should establish a connection to Redis and maintain said connection.

Screenshots

iwexPfQuHc

ZCDKnBTyHY

Logs

authentik-worker-logs.txt authentik-server-logs.txt

The Redis logs say that they are accepting the connection.

1:M 20 Nov 2024 22:18:17.564 - Accepted 172.18.1.2:35926
1:M 20 Nov 2024 22:18:17.792 - Accepted 172.18.1.1:59372
1:M 20 Nov 2024 22:18:17.890 - Accepted 172.18.1.1:59388
1:M 20 Nov 2024 22:18:17.890 - Accepted 172.18.1.1:59400

Version and Deployment (please complete the following information):

Additional context Tried on both 2024.10.2 and 2024.6.4

In the authentik-worker logs, it says that Redis connection was unsuccessful, however, if you immediately restart, then you see:

INF | event=Redis Connection successful logger=authentik.lib.config timestamp=1732174298.357012 

Makes zero sense how it can connect, and then can't. And other services are fine. And then after a few seconds of a successful Authentik / Redis connection, it will throw

warning | event=consumer: Connection to broker lost. Trying to re-establish the connection... exception=[{"exc_type":"ConnectionError","exc_value":"Connection closed by server.","frames":[{"filename":"/ak-root/venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py","lineno":340,"locals":{"blueprint":"<celery.worker.consumer.consumer.Consumer.Blueprint object at 0x74c8606a9340>","connection_retry":"True","connection_retry_type":"'broker_connection_retry'","exc":"ConnectionError('Connection closed by server.')","is_connection_loss_on_startup":"False","recoverable_errors":"\"(<class 'amqp.exceptions.ConnectionError'>, <class 'kombu.exceptions.Inconsisten\"+245","self":"<Consumer: celery@worker (running)>"},"name":"start"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/celery/bootsteps.py","lineno":116,"locals":{"i":"7","parent":"<Consumer: celery@worker (running)>","self":"<celery.worker.consumer.consumer.Consumer.Blueprint object at 0x74c8606a9340>","step":"<step: event loop>"},"name":"start"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py","lineno":746,"locals":{"c":"<Consumer: celery@worker (running)>","self":"<step: event loop>"},"name":"start"},{"filename":"/ak-root/venv/lib/python3.12/site-packages/celery/worker/loops.py","lineno":97,"locals":{"RUN":"1","blueprint":"<celery.worker.consumer.consumer.Consumer.Blueprint object at 0x74c8606a9340>","clock":"<LamportClock: 225>","connection":"<Connection: redis://:**@rs01:6379/0 at 0x74c85f81cad0>","consumer":"'<Consumer: 

Use Iptables for Firewall, which have always worked, and ports are properly whitelisted. Just to ensure it wasn't firewall, I turned the firewall all the way off, and removed any chance of any application preventing connection. But again, if I try connecting to Redis with any other container, or if I try to connect to Redis using Redis Insight, works fine.