Retrieving OAuth 2.0 tokens via Out Of Band `redirect_uri` is deprecated

[EmperorDali]

I'm not very well versed in OAuth 2.0, but it looks like Google has changed how part of their OAuth 2.0 flow works. We'll need to update how we obtain & refresh authorization tokens in order to keep functioning.

How this impacts org-gcal

Since

1. org-gcal uses the oob redirect_uri scheme to get authentication tokens, and,
2. this scheme has been deprecated, it will no longer be possible to use org-gcal.

This is already the case for any new OAuth 2.0 credentials you create (so don't delete them!). Existing credentials will continue to work through October 2022, at which point they will also cease to work.

Details

Here's my understanding of the situation. I think this correctly captures the major details, but my OAuth 2.0 terminology might not be quite right:

• Currently org-gcal uses the Out Of Band scheme to request OAuth 2.0 tokens, e.g., redirect_uri = urn:ietf:wg:oauth:2.0:oob. This means that org-gcal-request-authorization opens a browser window to an OAuth 2.0 flow that ends with a token code that can be pasted back into an emacs input prompt
• Google has deprecated the Out of Band scheme for OAuth 2.0 credentials. As of Feb 28 2022 newly created OAuth 2.0 credentials that attempt to use the oob redirect_uri fail with an invalid request error. Existing OAuth 2.0 credentials will continue to work with the oob redirect_uri scheme until the end of October 2022, at which point they will also cease working.
• According to the blog post linked right above, the recommended fix is to switch to the localhost redirect_uri scheme. This involves:
  • launch a local webserver on port XXXX prior to browsing to the OAuth 2.0 endpoint
  • send the OAuth 2.0 request with redirect_uri=http://localhost:XXXX
  • after clicking through Google's OAuth 2.0 flow, the user's browser will redirect to localhost:XXXX with the OAuth 2.0 token passed as a URL query parameters

Fixing this

I poked around for a few hours tonight at trying to fix this. My idea so far is rework org-gcal-request-authorization to start an http server, then parse the response out after the redirect and continue as normal.

We can create the server using simple-httpd and manage the callbacks using aio (which is what @telotortium settled on using for the other async code in #160)

This could look something like this - note that all "Desktop Application" OAuth 2.0 credentials created support the localhost redirect_uri.

(aio-defun org-gcal-request-authorization ()
  (let* ((gcal-auth-url
          (concat org-gcal-auth-url
                  "?client_id=" (url-hexify-string (funcall org-gcal-client-id))
                  "&response_type=code"
                  "&redirect_uri=http://127.0.0.1:1035" ;(url-hexify-string "urn:ietf:wg:oauth:2.0:oob")
                  "&scope=" (url-hexify-string org-gcal-resource-url)))
     (httpd-port 1035)
     (httpd-serve-files nil)
     (google-redirect-aio-callback (aio-make-callback))
     (callback (car google-redirect-aio-callback))
     (future (cdr google-redirect-aio-callback)))
    (cl-letf (((symbol-function 'httpd/) callback))
    (httpd-start)
    (browse-url gcal-auth-url)
    (let
    ((response (aio-await future)))
      (httpd-stop)
      (assoc "code" (nth 2 (cdr response)))
    ))))

Then

(aio-with-async (org-gcal-request-authorization))

returns a promise that resolves to the OAuth 2.0 code.

I'm happy to keep working on this but wouldn't mind suggestions from others on how to implement it. In particular, this aio-defun would need to interface with the deferred code in org-gcal-request-token. Maybe this means that this issue should be fixed on top of #160?

Possibly related issues: #97 #171 #173

[telotortium]

Hi, I'll try to get to this once I fix #160

[rhaps0dy]

For what it's worth, I've been writing and debugging an OAuth2 library using a (very simple) HTTP server in Emacs and Aio. Here it is https://github.com/rhaps0dy/emacs-oauth2-auto/

I think the interface for it is pretty good: a single function gets you an updated and refreshed token. I've been able to use it for org-gcal by adding to my config:

(defun org-gcal--get-access-token ()
  (oauth2-auto-access-token-sync "adria.garriga@mail-google.example.com" 'google))

then the oauth2 credentials for the apps that are at google are globally defined. This can definitely improve. But I spent a lot of time debugging this, so I definitely think it's worth incorporating its HTTP oauth2 mechanism into org-gcal, rather than coding it up from scratch.

As a bonus, it supports multiple accounts!

PS: Thank you very much telotortium for fixing and merging PR 192 !! (not linking on purpose)

[EmperorDali]

For what it's worth, I've been writing and debugging an OAuth2 library using a (very simple) HTTP server in Emacs and Aio. Here it is https://github.com/rhaps0dy/emacs-oauth2-auto/

Do you think your web-based OAuth2 functionality could be merged in to this existing emacs oauth2 package? It has the same problem where it uses the oob flow and thus will no longer work with Google. This is the package org-caldav uses for OAuth2 and so has this same problem with syncing against Google not working

If your feature was merged into emacs' oauth2 package (and we switched org-gcal over to using this package) both org-gcal and org-caldav would be fixed.

[rhaps0dy]

Do you think your web-based OAuth2 functionality could be merged in to this existing emacs oauth2 package?

Probably yes. It would replace almost all of the code, and I'm not sure the interface for this library is fully compatible with the functionality of oauth2, but it should be OK.

How do I even contribute to oauth2? Email the authors?

The part about having predefined authentication backends scratches my own itch and I'll keep it though. It's probably also good to keep it for other users; it's non-trivial and annoying to fill out all the parameters of oauth2-auth.

[EmperorDali]

Probably yes.

That would be super cool!

How do I even contribute to oauth2? Email the authors?

It looks like the oauth2 repo is on Savannah but it doesn't look like there's an obvious web-based way to submit PRs.

Probably your best bet would be to try emailing the maintainer Julien Danjou julien@danjou.info and see how he'd like to go about merging your PR. The library itself has been around a while, but it looks like Julien last commited to the repo in 2021 so it appears stable and actively maintained.

It would replace almost all of the code, and I'm not sure the interface for this library is fully compatible with the functionality of oauth2, but it should be OK.

FWIW I think keeping the oob functionality around would be useful since some OAuth 2.0 providers other than Google will likely continue allowing this flow. In my mind adding the localhost flow as an additional option would be a killer feature.

[lwiechec]

hi, I am also facing the problem with OAuth2 authentication with my 'app' and I tried to use @rhaps0dy 's package; unfortunately, I am getting 'Something went wrong' right after I choose my GMail user:

(the Google API's app is running in Testing status, with External user type, with my Google used added to Test Users; in Credentials section I can see one created, with Client ID and Secret that I use).

Any help would be greatly appreciated!

[rhaps0dy]

@lwiechec oh no! What's your config like? Are you using my pasted function above to substitute org-gcal--get-access-token? Does ~/.emacs.d/oauth2-auto.plist exist?

[lwiechec]

@rhaps0dy thanks for reaching out! yes, I've overrriden org-gcal--get-access-token function but I cannot see ~/.emacs.d/oauth2-auto.plist created.

The setup I tried:

(require 'oauth2-auto)

(require 'plstore)
(setq plstore-encrypt-to "coquelicot408@gmail.com")

(setq oauth2-auto-google-client-id "[REDACTED]"
      oauth2-auto-google-client-secret "[REDACTED]")

(defun org-gcal--get-access-token ()
  (oauth2-auto-access-token-sync "coquelicot408@gmail.com" 'google))
;;; other org-gcal-fetch stuff
(org-gcal-fetch)

PS: when I click on 'Next' button in the screenshot above, I am getting this:

[rhaps0dy]

Hmm, yeah, that looks annoying. Maybe the URL that [the (browse-url …) bit in oauth2-auto.el] opens is malformed. Maybe change that to (message …) and check it out? Mine looks like this:

<https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fmail.google.com%2F%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar.events&client_id=alphanumeric0id.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A34371%2F&login_hint=adria.garriga%40gmail.com&response_type=code&response_mode=query&access_type=offline&state=jp0IwFXr&code_challenge=yYRnRe8j_b3ykYsqhEJ15YVopzABIw_enBDzBjB56Sk&code_challenge_method=S256>

Maybe you’ve given your app only calendar permissions while my package asks for calendar and email permissions by default. You can try removing the <https://mail.google.com/> bit in https://github.com/rhaps0dy/emacs-oauth2-auto/blob/main/oauth2-auto.el#L87.

Another thing that occurs to me is maybe you have to change the “#” in the calendar @.to a%23`, like this: @.`.

[the (browse-url …) bit in oauth2-auto.el] https://github.com/rhaps0dy/emacs-oauth2-auto/blob/main/oauth2-auto.el#L350

[lwiechec]

hi again, thanks for help!

when I print the URL, I am getting:

https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fmail.google.com%2F%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar.events&client_id=REDACTED&redirect_uri=http%3A%2F%2Flocalhost%3A34415%2F&login_hint=coquelicot408%40gmail.com&response_type=code&response_mode=query&access_type=offline&state=J3OEV-FC&code_challenge=WxmJpzu4P6siYWT5HrJjx3kZ2OsWPbH0UAyZVwCRYhQ&code_challenge_method=S256

which looks similar (no unencoded '#' anywhere). I have also enabled the GMail permission in the app. It still complains about the request being malformed:

[rhaps0dy]

I'm very confused, your URL with my client ID and email works perfectly. Here's a list of possible problems: https://support.google.com/websearch/thread/95249301/how-do-i-fix-the-server-cannot-process-the-request-because-it-is-malformed-it-should-not-be-retried?hl=en . Maybe try pasting the URL in a private window of your browser, or clearing the cookies? Is your Client ID of the form 121212121212-alphanumeric01.apps.googleusercontent.com ?

[lwiechec]

thanks for help - it seems that this was indeed my browser's fault. Running it in private window of Firefox made it authenticate and receive the token. Thanks a million!

I am still not able to sync the calendar but it's because of something else: there seem to be problem with curl:

REQUEST [debug] REQUEST
REQUEST [debug] Run: curl --silent --include --location --compressed --cookie /home/luke/.emacs.d/request/curl-cookie-jar --cookie-jar /home/luke/.emacs.d/request/curl-cookie-jar --write-out \n(:num-redirects %{num_redirects} :url-effective "%{url_effective}") --data-binary @- --request POST https://www.googleapis.com/oauth2/v3/token
REQUEST [debug] REQUEST--CURL-CALLBACK event = exited abnormally with code 60

REQUEST [debug] REQUEST--CURL-CALLBACK proc = #<process request curl>
REQUEST [debug] REQUEST--CURL-CALLBACK buffer = #<buffer  *request curl*>
REQUEST [debug] REQUEST--CURL-CALLBACK symbol-status = nil
REQUEST [debug] REQUEST--CALLBACK
REQUEST [debug] (buffer-string) =

(:num-redirects 0 :url-effective "https://www.googleapis.com/oauth2/v3/token")
REQUEST [debug] REQUEST-RESPONSE--CANCEL-TIMER
REQUEST [debug] -CLEAN-HEADER
REQUEST [debug] -CUT-HEADER
REQUEST [debug] error-thrown = (error . "exited abnormally with code 60
")
REQUEST [debug] data = nil
REQUEST [debug] symbol-status = error
REQUEST [debug] Executing error callback.
REQUEST [error] Error (error) while connecting to https://www.googleapis.com/oauth2/v3/token.
REQUEST [debug] Executing complete callback.

...which indicates some issue in request.el...

[rhaps0dy]

Great! I suggest you open a new issue with that, and give more information about:

• what emacs version you're using?
• what versions of other packages (e.g. request.el, but also others) are you using?
• what OS and cURL version (these are unlikely to matter)

etc. Just so people can reproduce your bug.

[rhaps0dy]

Update: I have reported this bug to the Emacs bug tracker (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54913) and patches are welcome, so at some point in the next month I'll submit a patch for oauth2 there!

[lwiechec]

...after enabling verbose logging from curl I could see that it was not trusting Google's API SSL certificate. This might have something to do with the fact that I have a hybrid config on my PC: the curl comes from Guix (little experimentation of mine).

After setting the correct CA bundle via CURL_CA_BUNDLE env var I do see that the access_token comes back from Google's authentication service.

After switching back to 'silent' mode of curl in request.el I was able to sync the calendar.

Thanks again @rhaps0dy for help!

[telotortium]

For what it's worth, I've been writing and debugging an OAuth2 library using a (very simple) HTTP server in Emacs and Aio. Here it is https://github.com/rhaps0dy/emacs-oauth2-auto/

I think the interface for it is pretty good: a single function gets you an updated and refreshed token. I've been able to use it for org-gcal by adding to my config:

(defun org-gcal--get-access-token ()
  (oauth2-auto-access-token-sync "adria.garriga@mail-google.example.com" 'google))

then the oauth2 credentials for the apps that are at google are globally defined. This can definitely improve. But I spent a lot of time debugging this, so I definitely think it's worth incorporating its HTTP oauth2 mechanism into org-gcal, rather than coding it up from scratch.

As a bonus, it supports multiple accounts!

PS: Thank you very much telotortium for fixing and merging PR 192 !! (not linking on purpose)

@rhaps0dy I've made #200 to use oauth2-auto for org-gcal's OAuth2 authentication. Could you add oauth2-auto to Melpa?

[morphykuffour]

Is there a way to authorize Google Calendar and org-gcal. I followed the steps in the readme file and ended up on this thread. I checked out emacs-oauth2-auto but that was a mote point as his readme is not really clear. Is there an updated document on how to install org-gcal

[danielkrajnik]

Is this still an issue? Is it possible to use gcal to sync with org mode? If not, are there any workarounds?

[maikol-solis]

Is this still an issue? Is it possible to use gcal to sync with org mode? If not, are there any workarounds?

At least for me, the old API is still working. I included org-gcal as a testing app before October 2022, though.

Once #200 is merged, I can test it further.

Best.

[telotortium]

Hi everyone, I've merged #200 to switch to the oauth2-auto library, which uses the new API. Please pull the latest master (which is at tag v0.4.0) and confirm that everything works.

[danielkrajnik]

Fantastic, thank you! 😀

[maikol-solis]

Thanks!