Likely Dumb Newbie Questions

[whitedavidp]

So I use an older version of TBird as my email client on the desktop. Don't want to update it as that will break a bunch of plugins I wrote and use daily... Until now, I have been getting by using "app passwords" for gmail and hotmail/outlook accounts. But this morning I get an email from MS saying that in the near future password sign ins are being discontinued (they why did I have to go through all that rigamarole a short while ago?) so I went looking and found this project which seems so very promising to someone in my condition.

Right now I am focused only on my hotmail/outlook account. I have done my best to configure things and am basically following the info the the example config file. But:

1. I am still not quite sure what I need for client_id and client_secret. I just do not think I have another source to take these from. I do use BlueMail on my Android. And I am not even sure it that app will work when MS forces these changes (right now I am using app password same as in TBird). So I have to get these values somehow. But I a not very sure how.
2. I have tried going to the Entra/Azure link provided but it will not let me have access. I am so new to this I have no idea why or what to do.

So I am stuck at this point. I am sure this is all newbie stuff. But to me it is way over my pay grade. Can anyone point me in the right direction?

Thanks

[whitedavidp]

I just realized that I might be able to install a modern TBird on a spare computer, let it self-configure and they try to get/use info from that.

I have it connecting and when I go into the Security->Passwords and show myself what is there, I am wondering if the URL (1st column) and Password (3rd column) are what I need for oauth2_scope and client_secret? I assume that the client_id is just my full login email user id (xxx@outlook.com.

Am I on the right track?

So when I put those values into place and try to download (via pop) a message I have just sent to myself, I no longer get errors or prompts. But the connection login takes a really long time and nothing is actually downloaded.

So I am still not in heaven. Any tips are appreciated. Thanks

[whitedavidp]

Ok a little more progress...

I found some client ids for TBird here. And could confirm the url's in the sample file as well.

Now I am still prompted to authorize account. So when I do this (external or not) I get an error to the extent I cannot use my personal account and must use a school or work account (neither of which do I have).

Any tips? Thanks

[simonrob]

Thanks for following up – glad you managed to make progress with setting things up.

Re: your latest message –Microsoft have recently for some unknown reason decided to change the scope values for OAuth 2.0 access. Could you try the following value for oauth2_scope (adding the equivalent for IMAP if you need that functionality):

https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.SendAsApp offline_access

[whitedavidp]

Hi and thanks so much for your help. I tried the suggested change and, sadly, no joy still. The proxy prompts me to authorize the account as expected. But when I do so, I get the exact, same result as shown above. This time, I also clicked the sign-in options link and was shown the following: Not at all sure what this means. But between this and the message given on the previous dialog, it seems that there is something to do with MS knowing my account is personal but I am taken to an authorize page that is for organizations.

I am feeling quite close now and learning enough to be dangerous. But this is still mostly Greek to me :-)

Thanks

[whitedavidp]

I did some more looking at the TBird code referenced above. I noticed a couple things:

"outlook.office365.com",
     [
       "login.microsoftonline.com",
-      "https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/POP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access",
+      "https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access",
     ],

and

[
     "login.microsoftonline.com",
-    [
-      "08162f7c-0fd2-4200-a84a-f25a4db0b584", // Application (client) ID
-      "TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82", // @see App registrations | Certificates & secrets
+    {
+      clientId: "9e5f94bc-e8a4-4e73-b8be-63364c29d753", // Application (client) ID
       // https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-v2-protocols#endpoints
-      "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
-      "https://login.microsoftonline.com/common/oauth2/v2.0/token",
-    ],
+      authorizationEndpoint:
+        "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
+      tokenEndpoint:
+        "https://login.microsoftonline.com/common/oauth2/v2.0/token",
+      redirectionEndpoint: "https://localhost",
+    },
   ],

And so I started trying stuff out. One thing I noted was that the redirection endpoint in the code is https: while that in the example config file (and thereby also in my own) was http: (no s). That got me a little further along. But still no joy.

Then I tried the client_id = 9e5f94bc-e8a4-4e73-b8be-63364c29d753 and the following for the scope: oauth2_scope = https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access And that actually ended up getting me to the point where I could finally authorize via the proxy via the normal-looking, MS routine.

So now on to the next challenge: actually getting inbound POP email. TBird periodically polls for new email on this account. Regardless of what follows, I seem to get prompted by the proxy to re-authorize every single time. And that seems strange and rather painful. Perhaps I am missing something?

And I cannot help but notice that sometimes, when I do try to authorize when prompted by the proxy, a window pops up middle of the screen and then disappears fairly quickly. Amazingly, I sometimes then see the following notification from the proxy: And that is quite gratifying. However, sometimes I also get a failed notification and that seems odd. I have also seen TBird put up its failure dialog: Even with all of this, I still have not received a test email I sent to myself at this account - although I just had a thought that perhaps it is trapped in the outlook spam filter and need to check that to be sure.

Thanks for any info/suggestions. Your time and help are much appreciated. I am not sure if you will want to update the project with my findings above. Best

[whitedavidp]

I just checked and the email I was expecting WAS trapped in the outlook spam filter (not sure why). So I reported it as not-spam and it got moved to the inbox. Despite this, the pattern described above remains the same and that email was never downloaded to my TBird client.

It appears that the pattern is:

1. Every time TBird tries to connect to outlook via the proxy, I am prompted to authorize by the proxy.
2. I have to use the proxy's menu twice each time. The first time always seems to fail and I am prompted again. The second time seems to succeed (in that I am not prompted again).
3. I see the above login failure dialog in TBird right after (or possibly congruent with) seeing the success notification from the proxy.
4. No email is download.

Thanks again!

[whitedavidp]

I decided to try and see if I can get any further using my gmail account on TBird. Getting setup went much faster. Once again, I used the TBird source code I found referenced above. And so my server entry is:

permission_url = https://accounts.google.com/o/oauth2/auth
token_url = https://oauth2.googleapis.com/token
oauth2_scope = https://mail.google.com/
redirect_uri = http://localhost
client_id = 406964657835-aq8lmia8j95dhl1a2bvharmfk3t1hgqj.apps.googleusercontent.com
client_secret = kSmqreRr0qwBWJgbf5Y-PjSU

Authorization started smoothly enough. But after the first authorization page, I got taken to this error: I looked again at my server config and noticed again that the redirect was not https. So I changed that and tried it all again. Sadly, I came to the same end.

This makes me wonder which browser is being invoked to show/process these windows? I presumed it was the Windows default (which in my case is Firefox). So since this is google, I decided to try again after changing the default browser to Chrome. Sadly, still no joy.

Tips/suggestions are much appreciated. Thanks!

[whitedavidp]

Silly/stoopid me! I enabled external auth mode and tried again. I was able to succeed. As warned, the URL returned to the browser created a connection error. But when I copied the URL and pasted it, I was in business! Now to see how often I get asked to authorize... And I guess I will try with Outlook again this way. Best!

Well, I tried external authorization on Outlook as well. It appears to produce the same, exact results as described above using the internal method. Rats!

[whitedavidp]

Perhaps a little more progress... I found here another scope value to try with Outlook:

oauth2_scope = https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/EWS.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access

This ended up with a different result in the authorization process and I actually got an email from MS saying "New app(s) have access to your data". Have not seen that before! Seems like progress. However, I still am seeing the same pattern of intermittent auth success/failure.

So I turned on debugging (had not thought of that before - what a dope I am) and checked the log. I am seeing:

2024-07-04 12:50:06,837: Caught exception while requesting OAuth 2.0 credentials for account xxx@outlook.com: URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1000)'))

Is there a certificate somewhere in the proxy? Or is it using one on my system somewhere that may be the problem?

Thanks!

[simonrob]

I'm afraid I can't replicate this. With a Hotmail account (which I believe is treated the same as an Outlook account), and using the new scope values and the more recent Thunderbird client ID that you have used above (with no client secret), I can successfully authenticate and access the account.

You should definitely not be prompted to authenticate by the proxy every time you use the account. I was just about to point to debug mode, but I see you've done this already as I wrote this, which is good. That error is encountered when retrieving the token details, and may well be linked to the other issues you've encountered. You could try installing certifi?

Re: the issues you mentioned above where Thunderbird (which supports OAuth 2.0 natively, by the way) seems to just hang and not retrieve messages: it's important to note that this client is not a good choice for testing because it tries hard to simplify the connection process to the point where it modifies what you enter (such as encrypted vs. unencrypted connection details), often giving the impression it is using the values you've given when in fact it is not. As a result, when investigating an issue like this, it is always best to use one of the methods outlined in the troubleshooting section of the readme so that you can be totally sure of what is actually happening.

[whitedavidp]

Thanks so much. I just noticed that at the top of the log file I see the following:

2024-07-03 07:39:29,353: Running in a packaged/frozen environment - imported SSL certificates fromcertifi``

And I am running the pre-made executable from the .zip file on Windows 10. And that maps to your comment as well. It is interesting that Outlook should complain, however, and Gmail does not. Either way, I have no idea how I might update certifi (or anything else python-wise on Windows. I am presuming that Python and its supporting packages are somehow wrapped-up inside the .exe file. But trying to open it with 7-zip shows only one entry (endpoint-rule-set-1.json) and I cannot view that but I can extract it and then look at its contents. Not sure I see anything promising in there.

Regarding my TBird. I actually still use TBird v45 as anything newer will kill all the many plugins I have written and use daily. That version does NOT seem to support oauth and hence my searching for and finding (to my great joy) your excellent project.

[whitedavidp]

Logically, it seems that if there is a problem with a cert, that cert must be inside the Windows executable. TBird is not using crypto to the proxy. Does that need to be rebuilt with later certifi? I find it hard to believe the faulty cert is on the Outlook side. Thanks

[whitedavidp]

Since I have another, cleaner/simpler Windows machine on my LAN, I decided to copy over my setup there and run it. When I use that proxy, it works just great! So this convinces me that the issue has to do with a certificate on my original machine, NOT within python or anywhere else.

I would rather run the proxy locally than over my LAN like this. Doing so simplifies things for my simple mind. So now I have to try and see how/if I can normalize the computer's certificates from the other machine to my local one. It is making my brain hurt :-(

Could the problem comes from encrypting the data that is stored in the config file as a result of the authorization process? Is there some way to test this by turning off that encryption?

Thanks

[whitedavidp]

Wow, I am beginning to think I can cure the common cold (if I work hard enough)...

Realizing that there was indeed a problem on my desktop when I discovered that my other Windows machine on the LAN worked fine, I decided to explore the differences in the stored certificates between the 2 machines - which are otherwise quite similar.

I used the following powershell script on each machine and then diff'ed the two results:

Get-ChildItem cert:\ -Recurse | Where-Object {$_ -is [System.Security.Cryptography.X509Certificates.
X509Certificate2] -and $_.NotAfter -lt (Get-Date)} | Add-Member ScriptProperty -Name IssuedTo -Value { $this.GetNameInfo
( 'SimpleName', $false) } -PassThru |  Add-Member ScriptProperty -Name IssuedBy -Value { $this.GetNameInfo( 'SimpleName'
, $true ) } -PassThru | Select-Object -Property IssuedTo,IssuedBy,FriendlyName,NotAfter | Format-Table -AutoSize > C:\tm
p\expired_certs.txt

This demonstrated that the machine having troubles had a bunch of certificates installed that were not found on the working machine. I have no idea how the certificate being used is selected. So, after fully backing up the troubled machine, I went through the diff's one-by-by and searched for each cert that was not matched on the working machine on the basis of the IssuedTo and NotAfter columns. I had to look in both user and machine certs to find them all but I managed to remove the ones that were not present on the working machine and proved with a final diff showing no changes that I had them in sync.

As soon as I ran the proxy and authorized the Outlook account, all is well - working perfectly as expected and seen on the other machine..

Thanks so much for all of your time, patience, and help with this.

[simonrob]

Excellent! Thanks for following up, and glad you managed to solve this.

[MarkSchutt]

I also got a CERTIFICATE_VERIFY_FAILED error during the initial authentication of my Hotmail account.

Caught exception while requesting OAuth 2.0 credentials for account name@hotmail.com: URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1000)')))

That error occurred while the script communicated with https://login.microsoftonline.com/. At the same time there was no problem to create and use the SSL/TLS IMAP connection to outlook.office365.com or the respective Google servers for some different Gmail account which is also used with email-oauth2-proxy. SSL/TLS was basically working well, just not for the connection with https://login.microsoftonline.com/.

With the given debug logs it was not possible to figure out which certificate caused the error. Using Wireshark did not reveal any hints about any used certificates at all. Although there was a successful SSL/TLS connection visible in the captured packets, including a successful SSL handshake negotiation. It looked like the used certificates were transmitted as part of the encrypted application data on that SSL/TLS stream and therefore not visible to Wireshark.

Updating 'certifi' and/or using options like SSL_CERT_FILE= did not solve the problem.

The problem was eventually solved, by explicitly downloading and using the certificate chain for https://login.microsoftonline.com/. Just by using Firefox to access that URL, then opening the page properties and saving the entire certificate chain as PEM file "stamp2-login-microsoftonline-com_chain.pem". This PEM file was then set with

SSL_CERT_FILE=stamp2-login-microsoftonline-com_chain.pem

and email-oauth2-proxy was able to use these certificates successfully for the initial OAuth2 authentication with https://login.microsoftonline.com/

But what is really happening here and causing the problem?

The certificate chain for https://login.microsoftonline.com/ contains 3 fully valid certificates. server certificate "stamp2.login.microsoftonline.com" intermediate CA certificate "DigiCert SHA2 Secure Server CA" root CA certificate "DigiCert Global Root CA"

But at the same time, there were also 2 of those 'DigiCert SHA2 Secure Server CA' certificates in the Windows certificate store. One certificate was already expired and the other one was current. And this seems to cause a problem for the used Python SSL/TLS components. It seems that Python's urllib.py/ssl.py or one of the binaries "_ssl.pyd", "libcrypto-3.dll" or "libssl-3.dll" do have a problem with expired certificates that are found in the Windows certificate stores. I can only guess here what happened. I believe Python finds the expired certificate before the current one and therefore the SSL/TLS certificate verifications fails. However, detailed debugging would be needed here to understand what is really happening.

Fortunately, those (expired) certificates which may be available in the Windows certificate stores can get overruled by using an additional SSL_CERT_FILE which points to the current certificates. The certificates from an SSL_CERT_FILE are loaded in addition to the certificates from the Windows stores (which may or may not contain expired certificates). When using a SSL_CERT_FILE and loading only the certificate chain for https://login.microsoftonline.com/, access to outlook.office365.com and https://mail.google.com/ stills works as well.

Eventually that entire problem can get reproduced by just doing.

import urllib.request as urllib2
req = urllib2.Request("https://login.microsoftonline.com/", headers={'User-Agent':'Mozilla/5.0'})
urllib2.urlopen(req)

It's not an email-oauth2-proxy bug, not at all. It seems to be a general problem with Python and/or just Python under Windows and how expired certificates are handled.

[whitedavidp]

That's great detailed info. Thanks!

Personally, I guess I am glad to have all those dead certificates off my systems. But it did take time and effort and maybe I am just a neat freak!

[simonrob]

Thanks for following up with this useful update. I'm glad you managed to isolate the issue (and absolve the proxy of blame). This reminded me of another certificate-related issue that has previously been reported on Windows. While the problem you're describing doesn't sound directly related, there may be other fixes that can be applied. In that previous case, for example, a minor tweak to the python certificate handling function was able to work around the issue.