SMTP certificate behaviour changed with 2023.8 forward

[joachimtingvold]

Describe the bug SMTP server configured with TLS fails server validation in 2023.8.*

To Reproduce Steps to reproduce the behavior:

1: Configure SMTP;

AUTHENTIK_EMAIL__HOST=smtp.foo.bar
AUTHENTIK_EMAIL__PORT=587
AUTHENTIK_EMAIL__USE_TLS=true
AUTHENTIK_EMAIL__USE_SSL=false
AUTHENTIK_EMAIL__TIMEOUT=10
AUTHENTIK_EMAIL__FROM='noreply@foo.bar'

2. Send mail from Authentik (f.ex. via dc exec authentik-worker ak test_email mail@foo.bar)
3. Watch it fail due to SSL validation

Expected behavior Email should be sent without issues.

Screenshots N/A

Logs

{"event": "Error sending email, retrying...", "exc": "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')", "level": "debug", "logger": "authentik.stages.email.tasks", "pid": 222, "timestamp": "2023-09-01T03:59:00.872432"}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/manage.py", line 31, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/authentik/stages/email/management/commands/test_email.py", line 36, in handle
    send_mail(message.__dict__, stage.pk)
  File "/usr/local/lib/python3.11/site-packages/celery/local.py", line 182, in __call__
    return self._get_current_object()(*a, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 411, in __call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/autoretry.py", line 60, in run
    ret = task.retry(exc=exc, **retry_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 720, in retry
    raise_with_context(exc or Retry('Task can be retried', None))
  File "/usr/local/lib/python3.11/site-packages/celery/app/autoretry.py", line 38, in run
    return task._orig_run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/authentik/stages/email/tasks.py", line 103, in send_mail
    raise exc
  File "/authentik/stages/email/tasks.py", line 73, in send_mail
    backend.open()
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 92, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

Version and Deployment (please complete the following information):

• authentik version: 2023.8.1
• Deployment: docker-compose

Additional context

Same configuration works fine in 2023.6.2.

Root CA for the SMTP-server is mounted in /etc/ssl/certs (both when trying 2023.6.2 and 2023.8.1).

[drpetersen]

Tried with the exact same settings,

$ grep -i email authentik.env 
AUTHENTIK_EMAIL__HOST=smtp.******
AUTHENTIK_EMAIL__PORT=587
AUTHENTIK_EMAIL__USERNAME=admin@******
# AUTHENTIK_EMAIL__PASSWORD: Set in .env
AUTHENTIK_EMAIL__USE_TLS=true
AUTHENTIK_EMAIL__USE_SSL=false
AUTHENTIK_EMAIL__TIMEOUT=10
AUTHENTIK_EMAIL__FROM=admin@******

Works for me, with authentik 2023.8.2.

[joachimtingvold]

Is the certificate of the SMTP server you're using self-signed?

[drpetersen]

No, it's from letsencrypt.

[joachimtingvold]

Could be related to self-signed vs. public certificate, then. The SMTP servers I use have self-signed certificates (from internal PKI).

However, it works in 2023.6.2, and not in 2023.8.1, nor 2023.8.2. I only have to change the image parameter between the two to get working/non-working scenarios.

This works:

[…]
services:
  […]
  authentik-server:
    image: ghcr.io/goauthentik/server:2023.6.2
    environment:
      AUTHENTIK_EMAIL__HOST: smtp.foo.bar
      AUTHENTIK_EMAIL__PORT: 587
      AUTHENTIK_EMAIL__USE_TLS: true
      AUTHENTIK_EMAIL__USE_SSL: false
      AUTHENTIK_EMAIL__TIMEOUT: 10
      AUTHENTIK_EMAIL__FROM: 'login.foo.bar <noreply+login@foo.bar>'
    volumes:
      […]
      - /some/path/self_signed_ca.pem:/etc/ssl/certs/foobar_ca.pem:ro
      - /some/path/self_signed_issuing_ca.pem:/etc/ssl/certs/foobar_issuing_ca.pem:ro
    […]

This does not:

[…]
services:
  […]
  authentik-server:
    image: ghcr.io/goauthentik/server:2023.8.2
    environment:
      AUTHENTIK_EMAIL__HOST: smtp.foo.bar
      AUTHENTIK_EMAIL__PORT: 587
      AUTHENTIK_EMAIL__USE_TLS: true
      AUTHENTIK_EMAIL__USE_SSL: false
      AUTHENTIK_EMAIL__TIMEOUT: 10
      AUTHENTIK_EMAIL__FROM: 'login.foo.bar <noreply+login@foo.bar>'
    volumes:
      […]
      - /some/path/self_signed_ca.pem:/etc/ssl/certs/foobar_ca.pem:ro
      - /some/path/self_signed_issuing_ca.pem:/etc/ssl/certs/foobar_issuing_ca.pem:ro
    […]

[drpetersen]

Ok, I see. At first glance, I'd check for a permissions issue (although that would not explain why it used to work with 2023.6.2). But I assume you did that already?

docker exec authentik-worker md5sum /etc/ssl/certs/foobar*.pem
docker exec authentik-worker ls -l /etc/ssl/certs/foobar*.pem
docker exec authentik-worker id

Can the authentik user access your cert files?

I am also mounting custom certificates into the authentik-worker container, and in order to grant access, I assigned my authentik user to the ssl-cert group (put user 1000:114 in the authentik-worker service in my compose file, where 114 is the numeric id of my ssl-cert group) and grant that group access to the cert files

chgrp ssl-cert -R /etc/ssl/certs
chmod g+rX -R /etc/ssl/certs

Sorry if this is too obvious, it's the only idea I have right now.

[joachimtingvold]

Both of the certificates are world-readable, but I just double checked that the authentik user indeed can read them;

root@docker1:~# docker exec authentik-worker runuser -u authentik -- md5sum /etc/ssl/certs/foobar_ca.pem
ed1[…]3cb  /etc/ssl/certs/foobar_ca.pem

root@docker1:~# docker exec authentik-worker runuser -u authentik -- md5sum /etc/ssl/certs/foobar_issuing_ca.pem
7a2[…]bc0  /etc/ssl/certs/foobar_issuing_ca.pem

[joachimtingvold]

Authentik uses stages/email/tasks.py, which again uses django/core/mail/backends/smtp.py. Since Authentik doesn't pass the ssl_certfile parameter, ssl.create_default_context() is used (as per this code). According to the documentation, this triggers SSLContext.load_default_certs(), which again triggers SSLContext.set_default_verify_paths(). The latter method is allowed to softfail, which might or might not happen in my scenario. And it's also somewhat unclear to me what paths/folders its actually attempting to use (i.e. if it's limited to only checking /etc/ssl/certs/ca-certificates.crt, that is most likely the culprit, as /etc/ssl/certs/ca-certificates.crt is not automatically updated with new certificates).

I did not require to mount these CA files in 2023.6.2 in order for SMTP TLS to work (even if the servers have self-signed certificates), so not sure why that even worked in 2023.6.2... I'm pretty sure this problem is caused by updating the underlying Debian image (to 12/Bookworm), which I assume happened in 2023.8.*.

Simply adding CA files to /usr/share/ca-certificates/ and/or /etc/ssl/certs/ isn't enough. Even running update-ca-certificates after adding the CA files is not enough (you have to explicitly define the certificate relative path under /usr/share/ca-certificates/ in /etc/ca-certificates.conf, and then run update-ca-certificates).

Doing a docker compose build specification, I was able to update the system CA store, which I can confirm resolves the issue;

docker-compose.yml:

services:
  […]
  authentik-server:
    […]
    image: ghcr.io/goauthentik/server:latest
    build: ./build/authentik
    […]
  authentik-worker:
    […]
    image: ghcr.io/goauthentik/server:latest
    build: ./build/authentik
    […]

./build/authentik/Dockerfile:

FROM ghcr.io/goauthentik/server:latest
USER root
RUN mkdir -p /usr/share/ca-certificates/foobar
COPY foobar_ca.pem /usr/share/ca-certificates/foobar/
COPY foobar_issuing_ca.pem /usr/share/ca-certificates/foobar/
RUN echo 'foobar/foobar_ca.crt' >> /etc/ca-certificates.conf
RUN echo 'foobar/foobar_issuing_ca.crt' >> /etc/ca-certificates.conf
RUN update-ca-certificates

It's a bit cumbersome to do custom build to resolve this. Maybe Authentik should be able to set the ssl_keyfile and ssl_certfile parameters when invoking the Django mail module (django/core/mail/backends/smtp.py), so that we are able to pass custom CA-chains/certificates to the underlying SSL module... like AUTHENTIK_EMAIL__SSL_CERTFILE and AUTHENTIK_EMAIL__SSL_KEYFILE or similar.

[authentik-automation[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[joachimtingvold]

The issue is still present. Something changed in 2023.8. that made this happen (probably Debian version change). I guess it's technically not a bug, but something that worked pre-2023.8. is no longer working. I guess this issue can be rephrased to implement support for setting ssl_keyfile and ssl_certfile via the Authentik configuration file.

[simevo]

this might be relevant: https://stackoverflow.com/a/78474038

[simevo]

I have a draft here, it is untested, very rough and on top of version/2024.8.3.

If it has a chance to be accepted as MR, I am willing to polish it and make it ready for submission!