search

curl / curl

A command line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, TFTP, WS and WSS. libcurl offers a myriad of powerful features
https://curl.se/
Other
35.38k stars 6.36k forks source link

SMTP Auth is not using XOAUTH2 even when specified with --login-options 'AUTH=XOAUTH2' \ #10259

Closed ndevln closed 7 months ago

ndevln commented 1 year ago

I did this

I want to send an E-Mail over SMTP using OAUTH2.

curl --ssl-reqd -v \
--url 'smtp://smtp.gmail.com' \
--user $USERNAME \
--login-options 'AUTH=XOAUTH2' \
--oauth2-bearer $ACCESSTOKEN \
--mail-from from@example.org \
--mail-rcpt to@example.org \
--upload-file mail.txt

For GMail this fails with 

< 250-smtp.gmail.com at your service, [2001:9e8:3c3:7c00:8d8:c156:791a:89a]
< 250-SIZE 35882577
< 250-8BITMIME
< 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
< 250-ENHANCEDSTATUSCODES
< 250-PIPELINING
< 250-CHUNKING
< 250 SMTPUTF8
} [5 bytes data]
> AUTH OAUTHBEARER
{ [5 bytes data]
< 334
} [5 bytes data]
> TOKEN
{ [5 bytes data]
< 334 eyJzdGF0dXMiOiJpbnZhbGlkX3JlcXVlc3QiLCJzY29wZSI6Imh0dHBzOi8vbWFpbC5nb29nbGUuY29tLyJ9 -> {"status":"invalid_request","scope":"https://mail.google.com/"}
} [5 bytes data]
> AQ==
{ [5 bytes data]
< 535-5.7.8 Username and Password not accepted. Learn more at
< 535 5.7.8  https://support.google.com/mail/?p=BadCredentials b6-20020aa7d486000000b0048447efe3fcsm2477734edr.84 - gsmt
curl: (67) Login denied

The bearer token is sent with AUTH OAUTHBEARER and formatted according to the format on line 72: https://github.com/curl/curl/blob/5a9a5e171e186da14dc9d209a17304ed03c6cac8/lib/vauth/oauth2.c#L68-L73

Which should be supported according to: < 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH But since I specified --login-options 'AUTH=XOAUTH2' this method should be used.

I expected the following

According to Google the bearer token should be sent with AUTH XOAUTH2 https://developers.google.com/gmail/imap/xoauth2-protocol

And this token format should be used: https://github.com/curl/curl/blob/5a9a5e171e186da14dc9d209a17304ed03c6cac8/lib/vauth/oauth2.c#L95-L100

Microsoft describes the same standard. https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth According to this site these are the main provider using SASL XOAUTH2 as the authentication mechanism: https://mailtrap.io/blog/smtp-auth/

Since I never got it working, I don't know if this is the reason for the login the problem. But curl should use the specified login mechanism.

Thank you for all your work.

curl/libcurl version

$ curl -V
curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/1.1.1s (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.3 libssh2/1.10.0 nghttp2/1.51.0
Release-Date: 2022-10-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe TLS-SRP UnixSockets zstd

AND

$version
[1] "7.64.1"

$ssl_version
[1] "(OpenSSL/1.1.1k) Schannel"

$libz_version
[1] "1.2.12"

$libssh_version
[1] "libssh2/1.9.0"

$libidn_version
[1] NA

$host
[1] "x86_64-w64-mingw32"

$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps"  "ldap"   "ldaps"  "pop3"   "pop3s" 
[14] "rtsp"   "scp"    "sftp"   "smtp"   "smtps"  "telnet" "tftp"  

$ipv6
[1] TRUE

$http2
[1] FALSE

$idn
[1] TRUE

operating system

Window 10 22H2 Build 19045.2364

polarathene commented 7 months ago

UPDATE: See my follow-up comment.

Compared to the reporter @ndevln they have cited two versions of curl they tested? 7.64.1 and 7.86.0

While 7.81.0 and 7.88.1 aren't working for me (XOAUTH2 is treated as OAUTHBEARER), I have found curl 7.74.0 does work correctly as shown below.

This is a weird one...

I have no issue with XOAUTH2 with curl from Debian 11 Bullseye, but have noticed newer versions of curl mistakenly use OAUTHBEARER on Debian 12 Bookworm and WSL2, however all these versions of curl are newer than was reported here, so something else must be contributing to that? 🤔

You can reproduce a working example I shared here, but use curl within the container.

This is running on Windows 10 23H2 Build 22631.3007, via WSL2 (Ubuntu):

# Start the `compose.yaml` example:
$ docker compose up -d

# Correctly uses XOAUTH2:
$ docker exec -it dms-mail \
  curl -sv --url 'smtp://localhost:587' \
  --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test \
  --mail-from john.doe@example.test --mail-rcpt jane.doe@example.test --upload-file - <<< 'Hello Jane!'

*   Trying 127.0.0.1:587...
* Connected to localhost (127.0.0.1) port 587 (#0)
< 220 mail.example.test ESMTP
> EHLO mail
< 250-mail.example.test
< 250-PIPELINING
< 250-SIZE 10240000
< 250-ETRN
< 250-AUTH PLAIN LOGIN OAUTHBEARER XOAUTH2
< 250-AUTH=PLAIN LOGIN OAUTHBEARER XOAUTH2
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250-DSN
< 250 CHUNKING
> AUTH XOAUTH2
< 334
> dXNlcj1qb2huLmRvZUBleGFtcGxlLnRlc3QBYXV0aD1CZWFyZXIgRE1TX1lXTmpaWE56WDNSdmEyVnUBAQ==
< 235 2.7.0 Authentication successful
> MAIL FROM:<john.doe@example.test>
< 250 2.1.0 Ok
> RCPT TO:<jane.doe@example.test>
< 250 2.1.5 Ok
> DATA
< 354 End data with <CR><LF>.<CR><LF>
} [12 bytes data]
< 521 5.5.2 mail.example.test Error: bare <LF> received
* Connection #0 to host localhost left intact

# Postfix log:
$ docker logs dms-mail | grep 'postfix/submission/smtpd.*sasl_method'
Jan 24 00:01:40 mail postfix/submission/smtpd[1011]: 03A4B1EAEC: client=localhost[127.0.0.1], sasl_method=XOAUTH2, sasl_username=john.doe@example.test

# Curl version info:
$ curl -V
curl 7.74.0 (x86_64-pc-linux-gnu) libcurl/7.74.0 OpenSSL/1.1.1w zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2020-12-09, security patched: 7.74.0-1.3+deb11u11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Debian 12 / WSL2 Ubuntu curl versions:

# WSL2
$ curl -V
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.14
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

# Debian 12:
$ docker run --rm -it debian:12-slim bash
$ apt-get update && apt-get install curl

$ curl -V
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.11 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2023-02-20, security patched: 7.88.1-10+deb12u5
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Same happens with IMAP auth:

$ curl -sv --url 'imap://localhost:143' \
  --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test \
  -X LOGOUT

*   Trying 127.0.0.1:143...
* Connected to localhost (127.0.0.1) port 143 (#0)
< * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN AUTH=OAUTHBEARER AUTH=XOAUTH2] Dovecot ready.
> A001 CAPABILITY
< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN AUTH=OAUTHBEARER AUTH=XOAUTH2
< A001 OK Pre-login capabilities listed, post-login capabilities have more.
> A002 AUTHENTICATE OAUTHBEARER bixhPWpvaG4uZG9lQGV4YW1wbGUudGVzdCwBaG9zdD1sb2NhbGhvc3QBcG9ydD0xNDMBYXV0aD1CZWFyZXIgRE1TX1lXTmpaWE56WDNSdmEyVnUBAQ==
< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA
< A002 OK Logged in
> A003 LOGOUT
< * BYE Logging out
< A003 OK Logout completed (0.001 + 0.000 secs).
* Connection #0 to host localhost left intact

# Dovecot log:
$ docker logs dms-mail | grep 'dovecot: imap-login: Login:.*method='
Jan 24 00:39:26 mail dovecot: imap-login: Login: user=<john.doe@example.test>, method=OAUTHBEARER, rip=172.19.0.1, lip=172.19.0.4, mpid=4539, session=<jkaKS6YPQsisEwAB>

There is nothing special about the Debian 11 container environment, I can reproduce the correct behaviour via a plain Debian 11 container with curl installed there too.

polarathene commented 7 months ago

Taking Debian out of the mix, there is versioned curl containers available on DockerHub (which saves me from building/installing curl release binaries myself).

# Run curl  via a separate container out of the compose.yaml
# NOTE: It needs to be connected to the network (docker network ls) compose created for connectivity:
$ docker run --rm -it \
  --network the_implicit_compose_network \
  curlimages/curl:7.74.0 \
    -sv --url 'imap://mail.example.test:143' \
    --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test  \
    -X LOGOUT

This shows the failure was introduced after 7.79.1 with 7.80.0 release, but quite a lot changed and I don't really want to do a git bisect to build and test for the relevant commit 😓

At a guess, perhaps this change had unintended consequences (no other commits in that range have an oauth / sasl keyword match): https://github.com/curl/curl/pull/6930

https://github.com/curl/curl/blob/2620aa930bc73af1e4c70b10e3125b957b96ecfb/lib/curl_sasl.c#L216-L217

https://github.com/curl/curl/blob/2620aa930bc73af1e4c70b10e3125b957b96ecfb/lib/curl_sasl.c#L363

Which somehow results in the OAUTHBEARER case being triggered (possibly only because it's first?):

https://github.com/curl/curl/blob/2620aa930bc73af1e4c70b10e3125b957b96ecfb/lib/curl_sasl.c#L456-L477

https://github.com/curl/curl/blob/2620aa930bc73af1e4c70b10e3125b957b96ecfb/lib/curl_sasl.c#L681-L697

jay commented 7 months ago

At a guess, perhaps this change had unintended consequences (no other commits in that range have an oauth / sasl keyword match): #6930

@monnerat

monnerat commented 7 months ago

At a guess, perhaps this change had unintended consequences

Yes, I found it: a part of this commit maps given http authentication options to sasl ones and this works fine in the library. However the cli tool sets the http login option CURLAUTH_BEARER as a side-effect of --oauth2-bearer, causing the current problem.

I suggest to not preset anymore the sasl flags with http ones, but rather use them as a default, i.e. use them only if no login options has been specified.

If this solution is satisfactory, I can have a PR for it very soon.

polarathene commented 7 months ago

If this solution is satisfactory, I can have a PR for it very soon.

Below should provide an easy to test reproduction environment.

One command to get it up and running, then test via local curl build like shown below should verify correct functionality? (at least regarding with mail servers?)

Reproduction - Commands

# Start the `compose.yaml` example:
$ docker compose up -d

# Verify correct auth method is chosen:
$ curl -sv --url 'imap://localhost:143' \
    --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test  \
    -X LOGOUT \
    | grep 'AUTHENTICATE XOAUTH2'

$ curl -sv --url 'smtp://localhost:587' \
  --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test \
  --mail-from john.doe@example.test --mail-rcpt jane.doe@example.test --upload-file - <<< 'Hello Jane! \
   | grep 'AUTH XOAUTH2'

# Repeat for OAUTHBEARER instead of XOAUTH2

Reproduction - Docker Compose config

# Normally this would provide TLS configured for both services, that's been omitted to keep the reproduction simple
services:
  # Quick and easy mailserver setup with Postfix + Dovecot for testing OAuth2 support
  dms:
    image: docker.io/mailserver/docker-mailserver:13.3
    container_name: dms-mail
    hostname: mail.example.test
    environment:
      # Enable the OAuth2 support in Dovecot and configure it for our mocked service (caddy):
      ENABLE_OAUTH2: 1
      OAUTH2_INTROSPECTION_URL: http://auth.example.test/userinfo/
    # Test authentication against these ports:
    ports:
      - "143:143" # IMAP STARTTLS (Dovecot)
      - "587:587" # SMTP STARTTLS (Postfix)
    configs:
      - source: dms-accounts
        target: /tmp/docker-mailserver/postfix-accounts.cf

  # This would normally be a proper auth service, this is sufficient to mock out the required behaviour for testing
  caddy-oauth2:
    image: caddy:2.7
    container_name: dms-oauth2
    # Leverage Docker's internal DNS for the private network bridge it creates between services:
    hostname: auth.example.test
    ports:
      - "80:80"
    configs:
      - source: mock-auth-service
        target: /etc/caddy/Caddyfile

# Using the Docker Compose `configs.content` feature instead of volume mounting separate files.
# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer:
# https://github.com/compose-spec/compose-spec/pull/446
configs:
  # Basic Caddyfile example, see a better documented equivalent at:
  # https://github.com/docker-mailserver/docker-mailserver/blob/v13.3.0/test/config/oauth2/Caddyfile
  mock-auth-service:
    content: |
      :80 {
        @auth header Authorization "Bearer DMS_YWNjZXNzX3Rva2Vu"

        handle @auth {
          respond `{ "email": "john.doe@example.test", "email_verified": true }`
        }
        # Otherwise fail when expected auth header and value were not matched:
        respond 401 {
          close
        }
      }

  # DMS expects an account to be configured to run, this config provides one
  # You can add new accounts with `docker compose exec dms setup email add user@example.test bad-password`
  # Login credentials:
  # user: "john.doe@example.test" password: "secret"
  # user: "jane.doe@example.test" password: "secret"
  dms-accounts:
    # NOTE: `$` needed to be repeated to escape it,
    # which opts out of the `compose.yaml` variable interpolation feature.
    content: |
      john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
      jane.doe@example.test|{SHA512-CRYPT}$$6$$o65y1ZXC4ooOPLwZ$$7TF1nYowEtNJpH6BwJBgdj2pPAxaCvhIKQA6ww5zdHm/AA7aemY9eoHC91DOgYNaKj1HLxSeWNDdvrp6mbtUY.