postfix/submissions/smtpd: SASL authentication failure: Couldn't find mech XOAUTH2

kduthoy commented 1 month ago

📝 Preliminary Checks

[X] I tried searching for an existing issue and followed the debugging docs advice, but still need assistance.

👀 What Happened?

I use roundcube to connect to IMAP with OAUTH2 and everything works correct. I can login to the mailbox and read the mails.

When I try to send an e-mail, I get the following error in the docker mailserver logs:

postfix/submissions/smtpd[1653]: warning: SASL authentication failure: Couldn't find mech XOAUTH2
postfix/submissions/smtpd[1653]: warning: SASL XOAUTH2 authentication failed: no mechanism available, sasl_username=(unavailable)

👟 Reproduction Steps

Authenticate with OAUTH2 and try to send and e-mail

🐋 DMS Version

v14.0.0

💻 Operating System and Architecture

Ubuntu 22.04

⚙️ Container configuration files

No response

📜 Relevant log output

postfix/submissions/smtpd[1653]: warning: SASL authentication failure: Couldn't find mech XOAUTH2
postfix/submissions/smtpd[1653]: warning: hostname[ip-adres]: SASL XOAUTH2 authentication failed: no mechanism available, sasl_username=(unavailable)

polarathene commented 1 month ago

You should share your compose.yaml like the bug report asks, did you customize any other ENV?

@thechubbypanda may have some insight here as they contributed the XOAUTH2 support and docs for Roundcube support. I think ports 465 and 587 should support XOAUTH2 since Postfix delegates SASL auth to Dovecot by default, but perhaps something is missing there?

Our test suite shows coverage for SMTP with port 587, so this should definitely be supported:

https://github.com/docker-mailserver/docker-mailserver/blob/34423c2f665b8c7839c40fbc7d1fe8cda6342456/test/tests/serial/mail_with_oauth2.bats#L94-L109

Must be a misconfiguration somewhere?

kduthoy commented 1 month ago

Here is my compose file

services:
  mail:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    container_name: mail
    hostname: <FQDN>
    networks:
      lan:
        ipv4_address: <internal IP address>
    env_file: mail.env
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS, Authentication is DISABLED => use port 465/587 instead)
      - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
    volumes:
      - ./data/mail/mail/:/var/mail/
      - ./data/mail/mail-state/:/var/mail-state/
      - ./data/mail/log/:/var/log/mail/
      - ./appdata/mail/config/:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - ./appdata/proxy/letsencrypt/:/etc/letsencrypt/:ro
    restart: always
    stop_grace_period: 1m
    # Uncomment if using `ENABLE_FAIL2BAN=1`:
    cap_add:
      - NET_ADMIN
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0
    depends_on:
      - ldap

and the env-file:

mail.env

```env DMS_DEBUG=0 LOG_LEVEL=info SUPERVISOR_LOGLEVEL= ONE_DIR=1 ACCOUNT_PROVISIONER=LDAP POSTMASTER_ADDRESS=postmaster@example.com ENABLE_UPDATE_CHECK=1 UPDATE_CHECK_INTERVAL=1d PERMIT_DOCKER=none TZ=Europe/Brussels NETWORK_INTERFACE= TLS_LEVEL= SPOOF_PROTECTION=1 ENABLE_SRS=0 ENABLE_OPENDKIM=1 ENABLE_OPENDMARC=1 ENABLE_POLICYD_SPF=1 ENABLE_POP3= ENABLE_CLAMAV=1 ENABLE_RSPAMD=0 ENABLE_RSPAMD_REDIS= RSPAMD_LEARN=0 RSPAMD_CHECK_AUTHENTICATED=0 RSPAMD_GREYLISTING=0 RSPAMD_HFILTER=1 RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE=6 ENABLE_AMAVIS=1 AMAVIS_LOGLEVEL=0 ENABLE_DNSBL=0 ENABLE_FAIL2BAN=1 FAIL2BAN_BLOCKTYPE=drop ENABLE_MANAGESIEVE= POSTSCREEN_ACTION=enforce SMTP_ONLY= SSL_TYPE=manual SSL_CERT_PATH=/etc/letsencrypt/live/npm-9/fullchain.pem SSL_KEY_PATH=/etc/letsencrypt/live/npm-9/privkey.pem SSL_ALT_CERT_PATH= SSL_ALT_KEY_PATH= VIRUSMAILS_DELETE_DELAY= POSTFIX_DAGENT= POSTFIX_MAILBOX_SIZE_LIMIT= ENABLE_QUOTAS=1 POSTFIX_MESSAGE_SIZE_LIMIT= CLAMAV_MESSAGE_SIZE_LIMIT= PFLOGSUMM_TRIGGER= PFLOGSUMM_RECIPIENT= PFLOGSUMM_SENDER= LOGWATCH_INTERVAL= LOGWATCH_RECIPIENT= LOGWATCH_SENDER= REPORT_RECIPIENT= REPORT_SENDER= LOGROTATE_INTERVAL=weekly POSTFIX_REJECT_UNKNOWN_CLIENT_HOSTNAME=0 POSTFIX_INET_PROTOCOLS=all DOVECOT_INET_PROTOCOLS=all ENABLE_SPAMASSASSIN=1 SPAMASSASSIN_SPAM_TO_INBOX=1 ENABLE_SPAMASSASSIN_KAM=0 MOVE_SPAM_TO_JUNK=1 MARK_SPAM_AS_READ=0 SA_TAG=2.0 SA_TAG2=6.31 SA_KILL=10.0 SA_SPAM_SUBJECT=***SPAM*** ENABLE_FETCHMAIL=0 FETCHMAIL_POLL=300 ENABLE_GETMAIL=0 GETMAIL_POLL=5 LDAP_START_TLS= LDAP_SERVER_HOST="ldap://ldap.example.com" LDAP_SEARCH_BASE="dc=example,dc=com" LDAP_BIND_DN="uid=mail,ou=Application,dc=example,dc=com" LDAP_BIND_PW="****************************" LDAP_QUERY_FILTER_USER="(&(objectClass=inetOrgPerson)(mail=%s))" LDAP_QUERY_FILTER_GROUP="(&(objectClass=inetOrgPerson)(mailGroupMember=%s))" LDAP_QUERY_FILTER_ALIAS="(&(objectClass=inetOrgPerson)(mailAlias=%s))" LDAP_QUERY_FILTER_DOMAIN="(|(mail=*@%s)(mailAlias=*@%s)(mailGroupMember=*@%s))" LDAP_QUERY_FILTER_SENDERS="(&(objectClass=inetOrgPerson)(|(mail=%s)(mailAlias=%s)(mailGroupMember=%s)))" DOVECOT_TLS= DOVECOT_USER_FILTER="(&(objectClass=inetOrgPerson)(uid=%n))" DOVECOT_PASS_ATTRS="uid=user,userPassword=password" DOVECOT_USER_ATTRS="=home=/var/mail/%{ldap:uid},=mail=maildir:~,=uid=5000,=gid=5000" DOVECOT_PASS_FILTER="(&(objectClass=inetOrgPerson)(mail=%u))" DOVECOT_MAILBOX_FORMAT=maildir DOVECOT_AUTH_BIND=no ENABLE_POSTGREY=1 POSTGREY_DELAY=300 POSTGREY_MAX_AGE=35 POSTGREY_TEXT="Delayed by Postgrey" POSTGREY_AUTO_WHITELIST_CLIENTS=5 ENABLE_SASLAUTHD=1 SASLAUTHD_MECHANISMS=rimap SASLAUTHD_MECH_OPTIONS=127.0.0.1 SASLAUTHD_LDAP_SERVER= SASLAUTHD_LDAP_BIND_DN= SASLAUTHD_LDAP_PASSWORD= SASLAUTHD_LDAP_SEARCH_BASE= SASLAUTHD_LDAP_FILTER= SASLAUTHD_LDAP_START_TLS= SASLAUTHD_LDAP_TLS_CHECK_PEER= SASLAUTHD_LDAP_TLS_CACERT_FILE= SASLAUTHD_LDAP_TLS_CACERT_DIR= SASLAUTHD_LDAP_PASSWORD_ATTR= SASLAUTHD_LDAP_AUTH_METHOD= SASLAUTHD_LDAP_MECH= SRS_SENDER_CLASSES=envelope_sender SRS_EXCLUDE_DOMAINS= SRS_SECRET= DEFAULT_RELAY_HOST= RELAY_HOST= RELAY_PORT=25 RELAY_USER= RELAY_PASSWORD= ENABLE_OAUTH2=1 OAUTH2_INTROSPECTION_URL=https://user:password@authelia.example.com/api/oidc/introspection OAUTH2_USERNAME_ATTRIBUTE=username OAUTH2_INTROSPECTION_MODE=post OAUTH2_FORCE_INTROSPECTION=yes ```

kduthoy commented 1 month ago

I've tested a bit more and I get a different error when I change mail.env to the following

ENABLE_OAUTH2=1
OAUTH2_INTROSPECTION_URL=https://authelia.example.com/api/oidc/userinfo

Error in the logfile when I try to login in roundcube is. So now I don't even can't login with OAUTH2

dovecot: auth: oauth2(user,IP,<*******>): oauth2 failed: Introspection failed: Username 'user' did not match 'user@example.com'

Thank you for the help!

thechubbypanda commented 1 month ago

That looks like an email as username issue. Essentially you need to tell one side or the other to use email as username. I think the docs mention it vaguely but if not, there's a dovecot setting in the Auth config somewhere you'll need to tweak. I'm not at my PC at the moment so I can't link things for you.

polarathene commented 1 month ago

TL;DR

dovecot: auth: oauth2(user,IP,<*******>): oauth2 failed: Introspection failed: Username 'user' did not match 'user@example.com'

Like @thechubbypanda pointed out, you just need those two to match, so either login with user@example.com or configure the username_attribute to username instead of email (default) like you had configured earlier 👍

Full response

That looks like an email as username issue

Yes, when Dovecot receives the access token, it will provide it to the configured endpoint for OAUTH2_INTROSPECTION_URL and if the token is valid it receives a JSON response with information about the user. email is default field in the response to compare to the login username (credentials to go with the login password token), these need to match IIRC.

The field can be changed, but the support via ENV is not yet added in DMS, it will be for v15 release. For now you would need to directly mount the OAuth2 Dovecot config instead of using ENV.

I think the docs mention it vaguely

I know it's been discussed, it may be documented in our tests or scripts, but I don't think we raised awareness in the docs. I never got around to fleshing out the OAuth2 docs page as I wanted to get the improved config management support in first. I now have time to push ahead with the LDAP rework, and OAuth2 will leverage that same approach, so it'll be handled for v15 👍

OAUTH2_USERNAME_ATTRIBUTE=username
OAUTH2_INTROSPECTION_MODE=post
OAUTH2_FORCE_INTROSPECTION=yes

So @kduthoy you seem to be aware of the config you need - ~~but as mentioned above, the ENV support you're trying to use here isn't available yet.~~

EDIT: Actually it should work for those settings (_except force_introspection_):

https://github.com/docker-mailserver/docker-mailserver/blob/34423c2f665b8c7839c40fbc7d1fe8cda6342456/target/dovecot/dovecot-oauth2.conf.ext#L1-L4

This is the current feature support:

https://github.com/docker-mailserver/docker-mailserver/blob/34423c2f665b8c7839c40fbc7d1fe8cda6342456/target/scripts/startup/setup.d/oauth2.sh#L8

It's the same as the current LDAP, if the file is missing a line for the ENV to match, it will not work. If you need to modify the file to add a new setting, there is not much benefit from trying to use ENV.

Instead you can still keep the change in compose.yaml without a separate config file if you like via the configs feature (this requires any Docker Compose release from 2024):

services:
  mail:
    configs:
      - source: oauth2-config
        target: /etc/dovecot/dovecot-oauth2.conf.ext

configs:
  oauth2-config:
    content: |
      introspection_url = https://auth.example.com/admin/oauth2/introspect
      introspection_mode = post
      username_attribute = username

You can probably just use the /userinfo endpoint instead of /introspect, just bring back your username_attribute = username ENV setting and you shouldn't need the custom config override like shown above?

kduthoy commented 1 month ago

Hi @polarathene @thechubbypanda

thank you for your feedback.

Unfortunately it doesn't work. If I use

OAUTH2_INTROSPECTION_URL=https://auth.example.com/api/oidc/userinfo
OAUTH2_USERNAME_ATTRIBUTE=username

I get this error in the logs:

dovecot: auth: Error: oauth2(user, IP,<*******>): oauth2 failed: Introspection failed: No username returned

and I can't login to IMAP with roundcube.

If I add

OAUTH2_INTROSPECTION_URL=https://clientid:secret@authelia.example.com/api/oidc/introspection
OAUTH2_USERNAME_ATTRIBUTE=username
OAUTH2_INTROSPECTION_MODE=post

I can login to IMAP with roundcube but I get the following error when I send an e-mail with SMTP on port 465 and 587:

postfix/submissions/smtpd[1653]: warning: SASL authentication failure: Couldn't find mech XOAUTH2
postfix/submissions/smtpd[1653]: warning: hostname[ip-adres]: SASL XOAUTH2 authentication failed: no mechanism available, sasl_username=(unavailable)

Thank you for your help! Kristof

thechubbypanda commented 1 month ago

Check that dovecote is requesting 3 scopes: openid, profile, and email.
Ensure that the username attribute is being returned along with the email. If the username is under sub you might have to change the attribute to that.
Try logging in with your email address instead and setting the attribute to email.

polarathene commented 1 month ago

Seems like /userinfo doesn't return the username field, while your /introspection endpoint does.

Not sure why that isn't compatible with XOAUTH2 on Postfix, it should be delegating to Dovecot 🤔

I don't have time to investigate a reproduction. Potentially next month, I have to tackle other tasks before I can spare more time towards OAuth2 support.

I have recently rewritten docs related to this, you might want to give that a look? It's not merged yet, but you can view the preview for the OAuth2 page here, notably it might be worth trying my verification advice:

The curl example shows OAUTHBEARER, you could try with XOAUTH2 but pay attention to the notes that state the curl example won't work in the DMS container curl due to XOAUTH2 bug.

kduthoy commented 1 month ago

Check that dovecote is requesting 3 scopes: openid, profile, and email.

Checked and this is correct. Requesting openid, profie and email in the roundcube config

$config['oauth_scope'] = 'email openid profile';

Ensure that the username attribute is being returned along with the email. If the username is under sub you might have to change the attribute to that.

I found out that the attribute in the userinfo endpoint of Authelia for username is preferred_username. Changed the configuration to

OAUTH2_INTROSPECTION_URL=https://authelia.example.be/api/oidc/userinfo
OAUTH2_USERNAME_ATTRIBUTE=preferred_username

And I can login again with roundcube to my mailbox. Sending e-mails remains the same. It is correct that in the introspection endpoint the username attribute is username.

Sending e-mails remains a problem.

postfix/submission/smtpd[1526]: connect from roundcube[IP]
postfix/submission/smtpd[1526]: Anonymous TLS connection established from roundcube[IP]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384
postfix/submission/smtpd[1526]: warning: SASL authentication failure: Couldn't find mech XOAUTH2
postfix/submission/smtpd[1526]: warning: roundcube[IP]: SASL XOAUTH2 authentication failed: no mechanism available, sasl_username=(unavailable)

polarathene commented 1 month ago

I finally got around to setting up Roundcube with an auth service (rauthy). It has no issues with sending mail from a user I logged in to roundcube with via OAuth2.

I used the /userinfo endpoint instead of the /introspect endpoint, but I have documented insights regarding concerns for /introspect here: https://github.com/docker-mailserver/docker-mailserver/issues/2713#issuecomment-2268187285

Thus your issue is not a bug in DMS from what I can tell, everything is ok on our end. I would need to know what you're doing differently to reproduce.

Update

I have setup via Authelia too and got this to work fine. I had a user john configured with email john.doe@authelia.test and tried with the /userinfo endpoint.

username_attribute = preferred_username:

mail dovecot: auth: oauth2(john.doe@authelia.test,172.16.42.6,<8Plc1fgenKesECoG>): oauth2 failed: Introspection failed: Username 'john.doe@authelia.test' did not match 'john'

When leaving it with the default username_attribute = email this does work and if you haven't configured postfix-accounts.cf / LDAP in DMS for Dovecot to know of the user, it'll fail with:

roundcube-1  | errors: <f8bddfeb> IMAP Error: Login failed for john.doe@authelia.test against mail.example.test from 172.16.42.11 (X-Forwarded-For: 172.16.42.1). AUTHENTICATE XOAUTH2: A0001 NO [UNAVAILABLE] Internal error occurred. Refer to server log for more information. in /var/www/html/program/lib/Roundcube/rcube_imap.php on line 211 (GET /index.php/login/oauth?code=authelia_ac_UY9A8II1doCGPczbyUm5u3po2HuDS-KnzW4oDf4P85o.SIJOQ0lqEVBQLItSr-DoFlMu22nU_OaokmfM0uPNJhY&iss=https%3A%2F%2Fauth.example.localhost&scope=email+openid+profile&state=ejOw84Evfs4L)
dms-1        | 2024-08-06T00:06:09.986793+00:00 mail dovecot: auth: passwd-file(john.doe@authelia.test,172.16.42.6,<F0eWj/geiLasECoG>): unknown user (SHA1 of given password: 564053)
dms-1        | 2024-08-06T00:06:09.986824+00:00 mail dovecot: auth: Error: passwd-file(john.doe@authelia.test,172.16.42.6,<F0eWj/geiLasECoG>): user not found from userdb
dms-1        | 2024-08-06T00:06:09.986892+00:00 mail dovecot: imap(789): Error: auth-master: login: request [1902379009]: Login auth request failed: Authenticated user not found from userdb, auth lookup id=1902379009 (auth connected 0 msecs ago, request took 0 msecs, client-pid=786 client-id=1)
dms-1        | 2024-08-06T00:06:09.987189+00:00 mail dovecot: imap-login: Disconnected: Internal login failure (pid=786 id=1): user=<john.doe@authelia.test>, method=XOAUTH2, rip=172.16.42.6, lip=172.16.42.5, mpid=789, session=<F0eWj/geiLasECoG>

For /introspect endpoint, you could query this with curl and the access token (it's visible in the above log output as the value of code)

# This is only here to minimize noise from the endpoint URL that follows.
# It's the client ID + client Secret with `:` between the values
export CLIENT_CREDENTIALS=roundcube:JKGW34FcEJTF2MH9MqMMCELyaqIqoeJPIYpBwikbZUufTwKEpqeYpd6suuqwEUGQ

curl --request POST \
  --url "https://${CLIENT_CREDENTIALS}@auth.example.localhost/api/oidc/introspect" \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data token=authelia_ac_UY9A8II1doCGPczbyUm5u3po2HuDS-KnzW4oDf4P85o.SIJOQ0lqEVBQLItSr-DoFlMu22nU_OaokmfM0uPNJhY

{
  "active": true,
  "client_id": "roundcube",
  "exp": 1722906685,
  "iat": 1722903084,
  "scope": "email openid profile",
  "sub": "c09c699f-4f25-4a34-9103-46f6e23e762c",
  "username": "john"
}

Likewise with the /userinfo:

# This endpoint is protected via client credentials, just provide the access token via `Authorization: Bearer ...` header:

curl --request GET \
  --url http://auth.example.localhost/api/oidc/userinfo \
  --header 'Authorization: Bearer authelia_ac_UY9A8II1doCGPczbyUm5u3po2HuDS-KnzW4oDf4P85o.SIJOQ0lqEVBQLItSr-DoFlMu22nU_OaokmfM0uPNJhY

{
  "amr": [
    "pwd"
  ],
  "aud": [
    "roundcube"
  ],
  "auth_time": 1722901191,
  "azp": "roundcube",
  "client_id": "roundcube",
  "email": "john.doe@authelia.test",
  "email_verified": true,
  "iat": 1722901927,
  "iss": "https://auth.example.localhost",
  "name": "John Doe",
  "preferred_username": "john",
  "sub": "c09c699f-4f25-4a34-9103-46f6e23e762c"
}

So you can see the difference between the two endpoint responses there and what you can map with Dovecots username_attribute setting.

Roundcube seems to provide the IMAP login username from the email associated to the Authelia user as shown above, so you need to match that.

In this case preferred_username (/userinfo) and username (/introspect) JSON fields are john so choosing those won't match successfully. It can potentially match successfully by using the Dovecot OAuth2 config setting username_format (_default is %Lu, lowercase the IMAP login username john.doe@authelia.test and expect to match as the full email address to username_attribute_).

If we changed that default to %Ln, this would only compare the local-part of the IMAP login username, thus john.doe against john... which would still fail in this case, but you can see how if the email local-part or username in Authelia were actually equivalent this would then work too.

Anyway, just use the /userinfo endpoint and the default email. Then DMS should also have in postfix-accounts.cf or LDAP a user with that same email address and you should have successful login on IMAP, as well as SMTP submission?

github-actions[bot] commented 1 week ago

This issue has become stale because it has been open for 20 days without activity. This issue will be closed in 10 days automatically unless:

a maintainer removes the meta/stale label or adds the stale-bot/ignore label
new activity occurs, such as a new comment

github-actions[bot] commented 1 day ago

This issue was closed due to inactivity.

docker-mailserver / docker-mailserver