search

UvA-FNWI / M365-IMAP

MIT License
70 stars 17 forks source link

Do you have a solution for Emacs' SMTP? #3

Closed simurgh9 closed 3 years ago

simurgh9 commented 3 years ago

I was able to download mail using your instructions and offlineimap but I use Emacs to send email. Do you have a way to authenticate while sending email using Emacs' smtpmail-send-it if someone knows the following,

oauth2_client_id
oauth2_client_secret
oauth2_request_url
oauth2_refresh_token

I know that there is not a built-in method because smtpmail-auth-supported only has (cram-md5 plain login).

simurgh9 commented 3 years ago

I figured it out. If anyone wants it, reopen this issue and I'll just put the couple of elisp functions I wrote here. Note that the first thing that needs to happen is include the https://outlook.office.com/SMTP.Send scope in the config.py of this repository.

ethan-leba commented 3 years ago

I'm interested in your solution to this @simurgh9 if you still have that lying around!

simurgh9 commented 3 years ago

I'll have to remind myself of the context here before I post anything since I don't want to mistakenly post anything private haha. Quite busy these days. Maybe around Christmas? Hopefully that is not too late.

ethan-leba commented 3 years ago

Sounds good, no rush! :)

averter commented 2 years ago

Any update on this @simurgh9 ? I am also trying to get mu4e to send emails via oauth2ms. The key problem seems to be that SMTPAuth is disabled. Do we need to install M365-IMAP for your solution to work? Thanks in advance.

simurgh9 commented 2 years ago

Before you mess with lisp, make sure you follow what I said before.

Note that the first thing that needs to happen is include the https://outlook.office.com/SMTP.Send scope in the config.py of this repository.

Here is the Elisp I have for this,

;; MS-XOAUTH functionality for sending mail
(defun 786:b64-encoded-token ()
  "Encode access token according to https://tinyurl.com/2s6ymrv6"
  (base64-encode-string
   (concat "user=" smtpmail-smtp-user "auth=Bearer "
           (786:retrieve-access-token) "") t))

(defun 786:retrieve-access-token ()
  "Get access token from POST's response from `786:post`.
The body for the POST is built in `786:build-body`."
  (gethash "access_token"
           (786:post
            "https://login.microsoftonline.com/common/oauth2/v2.0/token"
            (786:build-body "personal/ou"))))

(defun 786:post (url data)
  "Make POST request to url with body = data. Returns JSON as hashtable."
  (let ((url-request-method "POST")
        (url-request-extra-headers `(("Content-Type" . "application/x-www-form-urlencoded")))
        (url-request-data data))
    (with-temp-buffer (url-insert-file-contents url)
        (json-parse-buffer :object-type 'hash-table))))

(defun 786:build-body (pass-entry-name)
  "Build the POST request body for SMTP authentication using password manager pass."
  (let ((entry-data (auth-source-pass-parse-entry pass-entry-name)))
    (concat "client_id=" (cdr (assoc "id" entry-data))
            "&client_secret=" (cdr (assoc "secret" entry-data))
            "&refresh_token=" (cdr (assoc "refresh_token" entry-data))
            "&grant_type=refresh_token")))

Then you gotta let smtpmail know about this,

(use-package smtpmail
      :ensure t
      :custom
        (epg-pinentry-mode 'loopback)
        (smtpmail-smtp-service 587)
        (smtpmail-stream-type  'starttls)
        (smtpmail-smtp-user "<your_email>") ; FIXME
        (smtpmail-smtp-server "smtp.office365.com")
        (smtpmail-default-smtp-server "smtp.office365.com")
        (auth-sources '("~/.config/emacs/.authinfo.gpg"))
      :config
        (add-to-list 'smtpmail-auth-supported 'xoauth2)
        (cl-defmethod smtpmail-try-auth-method
          (process (_mech (eql xoauth2)) user password)
          (let* ((access-token (786:b64-encoded-token)))
            (smtpmail-command-or-throw
             process (concat "AUTH XOAUTH2 " access-token) 235)))
        (setq message-send-mail-function 'smtpmail-send-it
              message-kill-buffer-on-exit t
              starttls-use-gnutls t))

You might want to change things in 786:build-body depending upon how you handle your refresh tokens etc.

averter commented 2 years ago

Thanks! I might be missing something obvious but, when cloning the repository it already contains that scope, as per the link of config.py that you shared. Strangely I had to add a (require 'auth-source-pass) before calling auth-source-pass-parse-entry (otherwise the function would be void) even though it is a built-in library. After integrating your functions this is the message log

auth-source-search: found 1 backends matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 0 results (max 1) matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 1 backends matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require nil :create nil)
auth-source-search: found 0 results (max 1) matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require nil :create nil)
530 5.7.57 Client not authenticated to send mail. [LO4P1967CA0153.USBD123.PROD.OUTLOOK.COM]
auth-source-search: found 0 CACHED results matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 1 backends matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require (:user :secret) :create t)
auth-source-search: found 0 results (max 1) matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require (:user :secret) :create t)
auth-source-search: found 0 CACHED results matching (:max 1 :host "smtp.office365.com" :port "587")
scroll-up-command: End of buffer
auth-source-search-backend: got 1 (max 1) in netrc:~/.config/emacs/.authinfo.gpg matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require (:user :secret) :create t)
auth-source-search: CREATED 1 results (max 1) matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require (:user :secret) :create t)
Contacting host: login.microsoftonline.com:443
221 2.0.0 Service closing transmission channel
mu4e-quote-for-modeline: https://login.microsoftonline.com/common/oauth2/v2.0/token: Bad Request

and the bad request token web link displays an error message with the following

Sign in

Sorry, but we’re having trouble with signing you in.
AKJST19251: The endpoint only accepts POST, OPTIONS requests. Received a GET request.

Any thoughts?

averter commented 2 years ago

This is really strange. I've checked and your POST function is being called, and so there is no apparent reason why GET is being used. This is my msmtprc

account work
host smtp.office365.com
from myactualemail@myworkplace
user myusername@myworkpalce
port 587
passwordeval "cd /home/myusername/M365-IMAP/; python3 refresh_token.py"
auth xoauth2
tls on
tls_starttls on
tls_certcheck on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log

and a log of the error

Debugger entered--Lisp error: (file-error "https://login.microsoftonline.com/common/oauth2/v2..." "Bad Request")
  signal(file-error ("https://login.microsoftonline.com/common/oauth2/v2..." "Bad Request"))
  url-http--insert-file-helper(#<killed buffer> "https://login.microsoftonline.com/common/oauth2/v2..." nil)
  url-insert-file-contents("https://login.microsoftonline.com/common/oauth2/v2...")
  (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table))
  (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (let ((url-request-method "POST") (url-request-extra-headers '(("Content-Type" . "application/x-www-form-urlencoded"))) (url-request-data data)) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))))
  786:post("https://login.microsoftonline.com/common/oauth2/v2..." "client_id=&client_secret=&refresh_token=&grant_typ...")
  (gethash "access_token" (786:post "https://login.microsoftonline.com/common/oauth2/v2..." (786:build-body "personal/ou")))
  786:retrieve-access-token()
  (concat "user=" smtpmail-smtp-user "�auth=Bearer " (786:retrieve-access-token) "��")
  (base64-encode-string (concat "user=" smtpmail-smtp-user "�auth=Bearer " (786:retrieve-access-token) "��") t)
  786:b64-encoded-token()
  (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235))
  (progn (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235)))
  (lambda (process _mech user password) (progn (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235))))(#<process smtpmail> xoauth2 "myusername@myworkpalce" "mypassword")
  apply((lambda (process _mech user password) (progn (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235)))) #<process smtpmail> xoauth2 ("myusername@myworkpalce" "mypassword"))
  smtpmail-try-auth-method(#<process smtpmail> xoauth2 "myusername@myworkpalce" "mypassword")
  smtpmail-try-auth-methods(#<process smtpmail> (8bitmime (auth login xoauth2) enhancedstatuscodes dsn (size \157286400)) "smtp.office365.com" 587 t)
  smtpmail-via-smtp(("myactualemail@myworkplace") #<buffer  smtpmail temp> t)
  smtpmail-via-smtp(("myactualemail@myworkplace") #<buffer  smtpmail temp>)
  smtpmail-send-it()
  message-multi-smtp-send-mail()
  message--send-mail-maybe-partially()
  message-send-mail(nil)
  message-send-via-mail(nil)
  message-send(nil)
  message-send-and-exit(nil)
  funcall-interactively(message-send-and-exit nil)
  call-interactively(message-send-and-exit nil nil)
  command-execute(message-send-and-exit)
averter commented 2 years ago

I've narrowed down one issue: my authinfo.gpg is in my home folder and therefore I have updated one line of your code from (auth-sources '("~/.config/emacs/.authinfo.gpg")) to (auth-sources '("~/.authinfo.gpg")) and now there are more chached results in the message log

Sending via mail...
auth-source-search: found 1 CACHED results matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 0 CACHED results matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require nil :create nil)
530 5.7.57 Client not authenticated to send mail. [FR0P281CA0148.DEUP281.PROD.OUTLOOK.COM]
auth-source-search: found 1 CACHED results matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 1 CACHED results matching (:host "smtp.office365.com" :port "587" :user "myusername@myworkpalce" :max 1 :require (:user :secret) :create t)
Contacting host: login.microsoftonline.com:443
221 2.0.0 Service closing transmission channel
mu4e-quote-for-modeline: https://login.microsoftonline.com/common/oauth2/v2.0/token: Bad Request

all except the 2nd call of auth-source-search seem to be working. Can someone share how the authinfo.gpg should look like? Mine has this structure

machine smtp.office365.com login myactualemail@myworkplace port 587 password mypassword
machine outlook.office365.com login myactualemail@myworkplace port 993 password mypassword
machine smtp.office365.com login myusername@myworkpalce port 587 password mypassword
machine outlook.office365.com login myusername@myworkpalce port 993 password mypassword
averter commented 2 years ago

Actually, I ought to have used my actual email. No issues with auth-source-search anymore. Message log now reads as

Sending via mail...
auth-source-search: found 1 backends matching (:max 1 :host "smtp.office365.com" :port "587")
Decrypting /home/myusername/.authinfo.gpg...done
auth-source-search-backend: got 1 (max 1) in netrc:~/.authinfo.gpg matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 1 results (max 1) matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 1 backends matching (:host "smtp.office365.com" :port "587" :user "myactualemail@myworkplace" :max 1 :require nil :create nil)
auth-source-netrc-parse: using CACHED file data for ~/.authinfo.gpg
auth-source-search-backend: got 1 (max 1) in netrc:~/.authinfo.gpg matching (:host "smtp.office365.com" :port "587" :user "myactualemail@myworkplace" :max 1 :require nil :create nil)
auth-source-search: found 1 results (max 1) matching (:host "smtp.office365.com" :port "587" :user "myactualemail@myworkplace" :max 1 :require nil :create nil)
Contacting host: login.microsoftonline.com:443
221 2.0.0 Service closing transmission channel
mu4e-quote-for-modeline: https://login.microsoftonline.com/common/oauth2/v2.0/token: Bad Request

just the bad request type at the end. Debugger is as follows. 

Debugger entered--Lisp error: (file-error "https://login.microsoftonline.com/common/oauth2/v2..." "Bad Request")
  signal(file-error ("https://login.microsoftonline.com/common/oauth2/v2..." "Bad Request"))
  url-http--insert-file-helper(#<killed buffer> "https://login.microsoftonline.com/common/oauth2/v2..." nil)
  url-insert-file-contents("https://login.microsoftonline.com/common/oauth2/v2...")
  (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table))
  (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (let ((url-request-method "POST") (url-request-extra-headers '(("Content-Type" . "application/x-www-form-urlencoded"))) (url-request-data data)) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (url-insert-file-contents url) (json-parse-buffer :object-type 'hash-table)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))))
  786:post("https://login.microsoftonline.com/common/oauth2/v2..." "client_id=&client_secret=&refresh_token=&grant_typ...")
  (gethash "access_token" (786:post "https://login.microsoftonline.com/common/oauth2/v2..." (786:build-body "personal/ou")))
  786:retrieve-access-token()
  (concat "user=" smtpmail-smtp-user "�auth=Bearer " (786:retrieve-access-token) "��")
  (base64-encode-string (concat "user=" smtpmail-smtp-user "�auth=Bearer " (786:retrieve-access-token) "��") t)
  786:b64-encoded-token()
  (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235))
  (progn (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235)))
  (lambda (process _mech user password) (progn (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235))))(#<process smtpmail> xoauth2 "myactualemail@myworkplace" "mypassword")
  apply((lambda (process _mech user password) (progn (let* ((access-token (786:b64-encoded-token))) (smtpmail-command-or-throw process (concat "AUTH XOAUTH2 " access-token) 235)))) #<process smtpmail> xoauth2 ("myactualemail@myworkplace" "mypassword"))
  smtpmail-try-auth-method(#<process smtpmail> xoauth2 "myactualemail@myworkplace" "mypassword")
  smtpmail-try-auth-methods(#<process smtpmail> (8bitmime (auth login xoauth2) enhancedstatuscodes dsn (size \157286400)) "smtp.office365.com" 587 nil)
  smtpmail-via-smtp(("myactualemail@myworkplace") #<buffer  smtpmail temp>)
  smtpmail-send-it()
  message-multi-smtp-send-mail()
  message--send-mail-maybe-partially()
  message-send-mail(nil)
  message-send-via-mail(nil)
  message-send(nil)
  message-send-and-exit(nil)
  funcall-interactively(message-send-and-exit nil)
  call-interactively(message-send-and-exit nil nil)
  command-execute(message-send-and-exit)

I feel I am getting closer (getting excited) :-)

averter commented 2 years ago

Before you mess with lisp, make sure you follow what I said before.

Note that the first thing that needs to happen is include the https://outlook.office.com/SMTP.Send scope in the config.py of this repository.

You might want to change things in 786:build-body depending upon how you handle your refresh tokens etc.

I've checked and the scope has been added on this commit 7f6620e0778bdf7ef0ebf32ab78e21875b9726ac. But what should be changed in the 786:build-body function? It might also be worthwhile to mention that I cannot enable SMTP Authentication or configure anything in Azure; the IT services at my workplace are a stubborn/unhelpful bunch. Any help is greatly appreciated.

averter commented 1 year ago

Oh I see...

  1. You are fetching the clientid, secret, etc. from pass and using them to create the body of POST. Unfortunately I don't use pass and don't care at this stage about keeping them encrypted, and so have copied them from config.py directly into your functions, e.g. "client_id=08162f7c-0fd2-4200-a84a-f25a4db0b584" "&client_secret=TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82"
  2. Your code is displaying some �, but I assume these are supposed to be ^A?

After applying the two above changes, I've tried once more and received this error message log (slightly edited)

Contacting host: login.microsoftonline.com:443
535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Tenant. Visit https://aka.ms/smtp_auth_disabled for more information. [BMLDERCRA15.sorprd10.prod.outlook.com]
221 2.0.0 Service closing transmission channel
smtpmail-send-it: Sending failed: 535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Tenant. Visit https://aka.ms/smtp_auth_disabled for more information. [BMLDERCRA15.sorprd10.prod.outlook.com] in response to AUTH

I'm starting to get the mechanics of this oauth2 thing, although I must say that this is certainly the product of a sick/twisted mind!

simurgh9 commented 1 year ago

Honestly, I do not understand this as well as you might think. But Authentication unsuccessful, SmtpClientAuthentication is disabled for the Tenant. is making me wonder that somewhere in your init, there is still an attempt of authentication using SMTP. As I understood it, we used SMTP package in lisp to setup the oxauth or what not. I'm sorry--I wish I could help more.

averter commented 1 year ago

I am really desperate to get this to work and any help/step that gets me closer to send emails is a huge help. In parallel to this I am also trying to receive/send emails via davmail, but unfortunately without much progress. I am using mu4e-contexts, and so it might be that some variable is not being overridden and the authentication is done via SMTP instead of xoauth2? Are you using mu4e-contexts by any chance? If so, have you set it up like this or should something be changed?

,(make-mu4e-context
  :name "WORK"
  :enter-func (lambda () (mu4e-message "Switch to the WORK context"))
  :leave-func (lambda () (setq mu4e-maildir-list nil))
  ;; we match based on the contact-fields of the message
  :match-func (lambda (msg)(when msg
                             (or (mu4e-message-contact-field-matches msg '(:to :bcc :cc) "myname@myworkplace")
                                 (string-prefix-p "/myname@myworkplace" (mu4e-message-field msg :maildir)))))
  :vars '((user-mail-address       . "myname@myworkplace" )
          (user-full-name          . "myname" )
          (mu4e-sent-folder        . "/myname@myworkplace/Sent")
          (mu4e-drafts-folder      . "/drafts")
          (mu4e-trash-folder       . "/myname@myworkplace/Trash")
          (mu4e-maildir-shortcuts .(("/myname@myworkplace/Inbox" . ?i);; setup some handy shortcuts
                                    ("/myname@myworkplace/Sent"  . ?s)
                                    ("/drafts"                    . ?d)
                                    ("/myname@myworkplace/Trash" . ?t)))
          ;; SMTP configuration
          (message-send-mail-function . smtpmail-send-it)
          (smtpmail-smtp-user . "myname@myworkplace")
          (smtpmail-stream-type . starttls)
          (smtpmail-smtp-server . "smtp.office365.com")
          (smtpmail-debug-info . t)
          (smtpmail-smtp-service . 587)))

Thanks in advance for any help.

averter commented 1 year ago

I was able to solve this by instead using davmail. Thanks for your help!