Sending failed: 501 Cannot Decode response

[dzfranklin]

I get the error smtpmail-send-it: Sending failed: 501 5.5.2 Cannot Decode response {{some lowercase letters and numbers}}.3 - gsmtp in response to AUTH XOAUTH2 {{some mixed case letters and numbers}}=

I'm omitting two parts because I guessed they wouldn't be helpful and might contain authentication information. I'm happy to share those two strings of random-looking characters if that isn't the case.

I believe the entirety of my relevant configuration is as follows. However, I'm new to Emacs, so I might have done something dumb elsewhere in my config file that's causing this error. In case this except is insufficient and you have the time to look at it I've uploaded my whole config file.

    (require 'smtpmail)
    (require 'auth-source-xoauth2)

    (setq message-send-mail-function 'smtpmail-send-it
          smtpmail-stream-type 'starttls
          smtpmail-default-smtp-server "smtp.gmail.com"
          smtpmail-smtp-server "smtp.gmail.com"
          smtpmail-smtp-service 587)

    ;; I use mu4e-contexts to set smtpmail-smtp-user to {{account1}} or {{account2}}

    (setq auth-source-xoauth2-creds #s(hash-table size 2 test equal
                                                  data (("smtp.gmail.com" "{{account1}}" "587")
                                                        (:token-url "https://accounts.google.com/o/oauth2/token"
                                                                    :client-id "{{client id}}"
                                                                    :client-secret "{{client secret}}"
                                                                    :refresh-token "{{refresh token1}}")

                                                        ("smtp.gmail.com" "{{account2}}" "587")
                                                        (:token-url "https://accounts.google.com/o/oauth2/token"
                                                                    :client-id "{{client id}}"
                                                                    :client-secret "{{client secret}}"
                                                                    :refresh-token "{{refresh token2}}"))))
    (add-to-list 'smtpmail-auth-supported 'xoauth2)
    (auth-source-xoauth2-enable)

I reuse the same client-id/client-secret pair and refresh tokens in offlineimap, where they work.

A minor note: I understand that storing secrets directly in a config file isn't recommended, but they're only backed up in an encrypted form and anyone with access to files on my computer would have access to my sign-in sessions in my browser.

[ccrusius]

This looks ok, but I have smtpmail-stream-sype set to 'ssl instead, and my server port set to 465. Could you try with those instead?

I also have smtpmail-debug-info and smtpmail-debug-verb set to t, as they can provide clues when things go wrong.

[dzfranklin]

I changed my smtpmail settings to be the same as yours, and set the debug variables as you suggested.

*trace of SMTP session to smtp.gmail.com*

220 smtp.gmail.com ESMTP l186sm4055530qkc.58 - gsmtp
250-smtp.gmail.com at your service, [129.133.215.70]
250-SIZE 35882577
250-8BITMIME
250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8
AUTH XOAUTH2 {{omitted}}=
501 5.5.2 Cannot Decode response l186sm4055530qkc.58 - gsmtp
QUIT
221 2.0.0 closing connection l186sm4055530qkc.58 - gsmtp

Process smtpmail deleted

*Messages*

Sending via mail...
501 5.5.2 Cannot Decode response l186sm4055530qkc.58 - gsmtp
221 2.0.0 closing connection l186sm4055530qkc.58 - gsmtp
smtpmail-send-it: Sending failed: 501 5.5.2 Cannot Decode response l186sm4055530qkc.58 - gsmtp in response to AUTH XOAUTH2 {{omitted}}=

[ccrusius]

Can you try with (setq auth-source-xoauth2-use-curl t)? Sometimes the built-in HTTP functions do not work with Gmail for some reason or another.

[dzfranklin]

Essentially the exact same error (difference response identifier and token only). Is there anything else to try?

[ccrusius]

A few things: try (setq gnutls-log-level 1), and try with only one account in your configuration.

The error message, "cannot decode," makes me believe what smtpmail is getting is an error message in plain English as a response to the AUTH XOAUTH2 command. If you could get that message somehow it would probably tell you what's going on.

[dzfranklin]

Thank you for all the help. Two areas of the logs stood out to me, and I would greatly appreciate your help in understanding them.

gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW
...
gnutls.c: [audit] Note that the security level of the Diffie-Hellman key exchange has been lowered to 256 bits and this may allow decryption of the session data

My intuition is that the likelihood Emacs is misunderstanding Gmail is greater than the likelihood Gmail is being insecure, and if this was really an issue Googling the error would lead to blog posts about it.

gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again. [195 times]

Could this mean that Google has flagged me as a potential spammer? Even if I raised the log level it wouldn't tell me what resource was unavailable. If retry I get generally between 60 and 200 attempts.

The full log output:

Sending via mail...
gnutls.c: [1] (Emacs) connecting to host: smtp.gmail.com
gnutls.c: [1] (Emacs) allocating credentials
gnutls.c: [1] (Emacs) setting the trustfile:  /etc/ssl/certs/ca-certificates.crt
gnutls.c: [1] (Emacs) gnutls callbacks
gnutls.c: [1] (Emacs) gnutls_init
gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW
gnutls.c: [1] (Emacs) setting the priority string
gnutls.c: [audit] Note that the security level of the Diffie-Hellman key exchange has been lowered to 256 bits and this may allow decryption of the session data

gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again. [195 times]
501 5.5.2 Cannot Decode response a2sm6177256qkl.71 - gsmtp
221 2.0.0 closing connection a2sm6177256qkl.71 - gsmtp
smtpmail-send-it: Sending failed: 501 5.5.2 Cannot Decode response a2sm6177256qkl.71 - gsmtp in response to AUTH XOAUTH2 {{omitted}}=

The full log output with log level 2 (I don't see anything useful in this, I'm posting it to avoid a roundtrip if you were going to ask for it): https://gist.github.com/danielzfranklin/7bbd099b90cbf7532e1097b0c1a95f62

[ccrusius]

Without seeing what the response that can't be decoded looks like, it is hard to tell what is going wrong. I remember getting some of those a long time ago, and figuring out what was wrong after I saw what they were (they were plain English error messages in my case). One way to go about it to see exactly what is happening (and I unfortunately have to do it every now and then when problems appear) is to step through the SMTP Lisp code in the Emacs debugger.

The Resource temporarily unavailable message is something that started a few months ago out of the blue. In my case, sending (or receiving) does not even work nowadays without the gnutls-log-level setting. It seems to be a race condition somewhere in Emacs, but I haven't filed a bug yet because I can't really tell where the problem is. I suspect the setting slows things down in the TLS layer enough to avoid triggering it.

Lastly, you don't mention the Emacs version you are using. I doubt it makes much of a difference, but I'm using 26.3.

[falloutphil]

@danielzfranklin - did you ever get to the bottom of your issue? If yes could you share the steps you took to diagnose/fix?

I had attempted almost exactly the same setup to you and found this page trying to google the issue I had... (I have now also changed my setup to exactly match that suggested @ccrusius above) and I get exactly the same error.

In the meantime, I'll have a poke around in the debugger as suggested to see if I can make more sense of it.

I am using emacs 26.3.

[dzfranklin]

@falloutphil I lost interest and never finished trying to debug. I'd appreciate it if you pinged me if you figured anything out. Good luck!

[falloutphil]

@danielzfranklin in my case I hadn't set smtpmail-smtp-user to my e-mail address.

This was being used by smtpmail-try-auth-methods to pass the username to auth-source-xoauth2--smtpmail-auth-method.

I used the debugger to inspect the value of the base64 created string and could see the neither user nor password were set. Using edebug I found I could see: smtpmail-command-or-throw(#<process smtpmail> "AUTH XOAUTH2 mYSecretString" 235)

When I ran mYSecretString through https://www.base64decode.org/ (if you are security conscious this might not be the best idea - instead it could be done programatically, I was fairly certain my password wasn't in there!) - it returned: user=
auth=Bearer



I wouldn't expect the password to be set - that's the point of xoauth, but the user in my case must be specified so it can look-up the xoauth details (keyed on user and smtp server address).

Working back over I arrived at smtpmail-smtp-user as the setting for the user.

Works flawlessly now.

Hope this helps!

[ccrusius]

Thanks @falloutphil for tracking your problem down. These interactions are quite hard to debug. The error messages really don't help.

Edit: in your case for example, the problem was that a variable that auth-source-xoauth2 dosen't even mention wasn't set. There are many layers to the whole process, and the reason why they don't work together is seldom clear.