goauthentik / authentik

The authentication glue you need.
https://goauthentik.io
Other
13.54k stars 905 forks source link

LDAP outpost does not start up #11196

Closed mxhash closed 2 months ago

mxhash commented 2 months ago

Describe the bug

After the upgrade to 2024.8.0 the ldap outpost does not start up and authentication is not possible.

Edit: Looks like the container tries to fetch users and groups. I left the container running for an hour, and it builds up memory until the kernel kills the process with an OOM (Out of Memory) error:

Screenshot 2024-09-04 at 09 56 45

To Reproduce Steps to reproduce the behavior:

  1. Pull containers to 2024.8.0
  2. Restart the stack

Expected behavior LDAP outpost starts the binder and the ports are open. Also the FE outpost status and the container should be healthy and authentication, or at least, bind with a service user should lead to success.

Logs

LDAP Outpost Container Logs:

ldap-1  | {"event":"Loaded config","level":"debug","path":"inbuilt-default","timestamp":"2024-09-04T05:35:59Z"}
ldap-1  | {"event":"Loaded config from environment","level":"debug","timestamp":"2024-09-04T05:36:00Z"}
ldap-1  | {"event":"not enabling debug server, set `AUTHENTIK_DEBUG` to `true` to enable it.","level":"info","logger":"authentik.go_debugger","timestamp":"2024-09-04T05:36:00Z"}
ldap-1  | {"event":"Fetched outpost configuration","level":"debug","logger":"authentik.outpost.ak-api-controller","name":"nethqauth1.ldap","timestamp":"2024-09-04T05:36:00Z"}
ldap-1  | {"event":"Fetched global configuration","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:36:00Z"}
ldap-1  | {"event":"HA Reload offset","level":"debug","logger":"authentik.outpost.ak-api-controller","offset":"9s","timestamp":"2024-09-04T05:36:00Z"}
ldap-1  | {"event":"Successfully connected websocket","level":"info","logger":"authentik.outpost.ak-ws","outpost":"0e3a3214-ba72-44d4-b0b5-8d1491a480c0","timestamp":"2024-09-04T05:36:01Z"}
ldap-1  | {"event":"Enabled USR1 hook to reload","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:36:01Z"}
ldap-1  | {"event":"Fetching certificate and private key","level":"info","logger":"authentik.outpost.cryptostore","timestamp":"2024-09-04T05:36:02Z","uuid":"6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4"}
ldap-1  | {"event":"Fingerprint hasn't changed, not fetching cert","level":"debug","logger":"authentik.outpost.cryptostore","timestamp":"2024-09-04T05:36:03Z","uuid":"6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4"}
ldap-1  | {"event":"initialised memory searcher","level":"debug","logger":"authentik.outpost.ldap.searcher.memory","timestamp":"2024-09-04T05:36:03Z"}

Outpost Requests on Server Container

server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/core/users/?include_groups=true", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 162, "remote": "172.22.0.5", "request_id": "6960488fcfab4908bcf7153a48768710", "runtime": 2568, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:37:51.839094", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "secret_key", "domain_url": "0.0.0.0", "event": "/api/v3/crypto/certificatekeypairs/6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4/", "host": "0.0.0.0:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 165, "remote": "127.0.0.1", "request_id": "03aefaaf1b4d423c8fd8885b0cebd403", "runtime": 303, "schema_name": "public", "scheme": "http", "status": 200, "timestamp": "2024-09-04T05:37:55.306156", "user": "ak-outpost-1f6970633084440d9a576a2ac960cdce", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "secret_key", "domain_url": "0.0.0.0", "event": "/api/v3/crypto/certificatekeypairs/6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4/", "host": "0.0.0.0:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "127.0.0.1", "request_id": "36e6e9ce83114d798ff286408b478566", "runtime": 280, "schema_name": "public", "scheme": "http", "status": 200, "timestamp": "2024-09-04T05:38:00.848321", "user": "ak-outpost-1f6970633084440d9a576a2ac960cdce", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/outposts/instances/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "be71b00b86a647b8a2c895aeaaf61182", "runtime": 169, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:01.119702", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/root/config/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "2f6f8fcf88834d9cac7b7af23b079bc9", "runtime": 57, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:01.273771", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "secret_key", "domain_url": "0.0.0.0", "event": "/api/v3/crypto/certificatekeypairs/6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4/", "host": "0.0.0.0:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "127.0.0.1", "request_id": "7cf77a26b4204469af28567b1b0c453e", "runtime": 293, "schema_name": "public", "scheme": "http", "status": 200, "timestamp": "2024-09-04T05:38:01.667046", "user": "ak-outpost-1f6970633084440d9a576a2ac960cdce", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/outposts/ldap/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "a9a7544c5099411bbeb9223d44e17a77", "runtime": 157, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:02.757149", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/core/brands/current/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "b8b8f20c20194e80b6d256c4faafe28e", "runtime": 82, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:02.923671", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/crypto/certificatekeypairs/6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "e0437d9ad40f42bca52c1460b4886998", "runtime": 348, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:03.356604", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/crypto/certificatekeypairs/6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4/view_certificate/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "66a7680cfd3a4ceabf6665b824bfdbe4", "runtime": 202, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:03.647293", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/crypto/certificatekeypairs/6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4/view_private_key/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "cea7f3c07cf34da49589c8095e21e528", "runtime": 209, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:03.942126", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}
server-1  | {"auth_via": "api_token", "domain_url": "server", "event": "/api/v3/crypto/certificatekeypairs/6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4/", "host": "server:9443", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 168, "remote": "172.22.0.5", "request_id": "435315e263cb4b61b1c591740aa10156", "runtime": 321, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-09-04T05:38:04.353146", "user": "ak-outpost-0e3a3214ba7244d4b0b58d1491a480c0", "user_agent": "goauthentik.io/outpost/2024.8.0"}

Successful startup of a 2024.6.4 LDAP outpost, but it does not accept the response from the server any more and no authentication is possible:

ldap-1  | {"event":"Loaded config","level":"debug","path":"inbuilt-default","timestamp":"2024-09-04T05:45:45Z"}
ldap-1  | {"event":"Loaded config from environment","level":"debug","timestamp":"2024-09-04T05:45:45Z"}
ldap-1  | {"event":"not enabling debug server, set `AUTHENTIK_DEBUG` to `true` to enable it.","level":"info","logger":"authentik.go_debugger","timestamp":"2024-09-04T05:45:45Z"}
ldap-1  | {"event":"Fetched outpost configuration","level":"debug","logger":"authentik.outpost.ak-api-controller","name":"nethqauth1.ldap","timestamp":"2024-09-04T05:45:46Z"}
ldap-1  | {"event":"Fetched global configuration","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:45:46Z"}
ldap-1  | {"event":"HA Reload offset","level":"debug","logger":"authentik.outpost.ak-api-controller","offset":"0s","timestamp":"2024-09-04T05:45:46Z"}
ldap-1  | {"event":"Successfully connected websocket","level":"info","logger":"authentik.outpost.ak-ws","outpost":"0e3a3214-ba72-44d4-b0b5-8d1491a480c0","timestamp":"2024-09-04T05:45:46Z"}
ldap-1  | {"event":"Enabled USR1 hook to reload","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:45:46Z"}
ldap-1  | {"event":"Fetching certificate and private key","level":"info","logger":"authentik.outpost.cryptostore","timestamp":"2024-09-04T05:45:48Z","uuid":"6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4"}
ldap-1  | {"event":"Fingerprint hasn't changed, not fetching cert","level":"debug","logger":"authentik.outpost.cryptostore","timestamp":"2024-09-04T05:45:50Z","uuid":"6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4"}
ldap-1  | {"event":"initialised memory searcher","level":"debug","logger":"authentik.outpost.ldap.searcher.memory","timestamp":"2024-09-04T05:45:50Z"}
ldap-1  | {"count":100,"event":"fetched users","level":"debug","page":1,"timestamp":"2024-09-04T05:45:52Z"}
ldap-1  | {"count":7,"event":"fetched users","level":"debug","page":2,"timestamp":"2024-09-04T05:45:53Z"}
ldap-1  | {"count":100,"event":"fetched groups","level":"debug","page":1,"timestamp":"2024-09-04T05:45:55Z"}
ldap-1  | {"count":27,"event":"fetched groups","level":"debug","page":2,"timestamp":"2024-09-04T05:45:56Z"}
ldap-1  | {"event":"initialised direct binder","level":"info","logger":"authentik.outpost.ldap.binder.direct","timestamp":"2024-09-04T05:45:56Z"}
ldap-1  | {"event":"Fingerprint hasn't changed, not fetching cert","level":"debug","logger":"authentik.outpost.cryptostore","timestamp":"2024-09-04T05:45:56Z","uuid":"6ff56ae8-84a0-4ebf-bbfe-90b4a4feb1e4"}
ldap-1  | {"event":"initialised memory searcher","level":"debug","logger":"authentik.outpost.ldap.searcher.memory","timestamp":"2024-09-04T05:45:56Z"}
ldap-1  | {"count":100,"event":"fetched users","level":"debug","page":1,"timestamp":"2024-09-04T05:46:00Z"}
ldap-1  | {"count":7,"event":"fetched users","level":"debug","page":2,"timestamp":"2024-09-04T05:46:00Z"}
ldap-1  | {"count":100,"event":"fetched groups","level":"debug","page":1,"timestamp":"2024-09-04T05:46:02Z"}
ldap-1  | {"count":27,"event":"fetched groups","level":"debug","page":2,"timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"initialised direct binder","level":"info","logger":"authentik.outpost.ldap.binder.direct","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Update providers","level":"info","logger":"authentik.outpost.ldap","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Starting Interval updater...","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Running interval update","level":"debug","logger":"authentik.outpost.ak-api-controller","loop":"interval-updater","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Starting WS Handler...","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Starting WS Health notifier...","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Starting LDAP SSL server","level":"info","listen":"0.0.0.0:6636","logger":"authentik.outpost.ldap","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Starting LDAP server","level":"info","listen":"0.0.0.0:3389","logger":"authentik.outpost.ldap","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Starting periodical timer...","level":"debug","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Starting Metrics server","level":"info","listen":"0.0.0.0:9300","logger":"authentik.outpost.metrics","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Pre-heating flow cache","flow":"default-authentication-flow","level":"debug","logger":"authentik.outpost.ldap","timestamp":"2024-09-04T05:46:03Z"}
ldap-1  | {"event":"Fetched outpost configuration","level":"debug","logger":"authentik.outpost.ak-api-controller","name":"nethqauth1.ldap","timestamp":"2024-09-04T05:46:04Z"}
ldap-1  | {"event":"Starting authentik outpost","hash":"tagged","level":"info","logger":"authentik.outpost","timestamp":"2024-09-04T05:46:04Z","version":"2024.6.4"}
ldap-1  | {"event":"initialised direct binder","level":"info","logger":"authentik.outpost.ldap.binder.direct","timestamp":"2024-09-04T05:46:12Z"}
ldap-1  | {"event":"initialised direct binder","level":"info","logger":"authentik.outpost.ldap.binder.direct","timestamp":"2024-09-04T05:46:18Z"}
ldap-1  | {"event":"Update providers","level":"info","logger":"authentik.outpost.ldap","timestamp":"2024-09-04T05:46:18Z"}

Version and Deployment (please complete the following information):

Additional context

It seems that the LDAP outpost hangs after fetching the certificate, and it does not start the binder. As a result, the ports do not open, making a connection impossible. I double-checked the object permissions, and a test with superuser privileges for the outpost user did not change anything. All requests respond with a status of 200. Simplifying the configuration and using vanilla documentation deployment examples were also unsuccessful.

The frontend shows the first health check request as OK (with no version), but then switches to ‘not available’.

Docker compose service:

  ldap:
    image: ghcr.io/goauthentik/ldap:2024.8.0
    restart: unless-stopped
    ports:
      - "6636:6636"
      - "9301:9300"
    environment:
        AUTHENTIK_HOST: https://server:9443
        AUTHENTIK_HOST_BROWSER: https://EXTERNAL_HOST
        AUTHENTIK_INSECURE: true
        AUTHENTIK_TOKEN: XXX
    #env_file:
    #  - .env
    depends_on:
      - server
cowboyxup commented 2 months ago

Having the same issue. However when i switched the Provider settings to direct binding and direct querying, the ldap starts and accepts searches, but is extremly slow and the memory usage of the ldap container skyrocket every minute about 100MB.

mxhash commented 2 months ago

Hi,

I can confirm that the startup was successful with the direct query and bind options configured. I’ve attached a screenshot showing the memory consumption for a single login. The login took more than 5 minutes.

ldap-outpost-ram

Cheers, Marius