DavMail 6.xx and 5.5.xx crash when using O365Interactive on OSX Sonoma 14.1.1 (23B81)

ifinkelstein commented 7 months ago

Thank you for this phenomenal project. I've been a happy DavMail/mbsync user for many years.

After upgrading to OSX Sonoma, DavMail crashes as soon as it enters the O365Interactive authentication frame. I cannot progress to an authentiation screen in manual mode for some reason. I can reproduce the crash in 6.xx and 5.5.xx.

I upgraded my Java to the latest version, but no luck. Importantly, DavMail was rock solid with the prior OS X release. This makes me think there is a problem between DavMail and OS X. DavMail log:

2023-11-14 08:41:47,823 DEBUG [main] davmail.DavGateway  - Start DavMail in GUI mode
2023-11-14 08:41:47,854 DEBUG [main] davmail  - OS Name: Mac OS X Java version: 1.8.0_391 64 System tray supported 
2023-11-14 08:41:47,854 INFO  [main] davmail  - O365Interactive is not compatible with SWT, do not try to create SWT tray
2023-11-14 08:41:47,975 INFO  [main] davmail  - DavMail Gateway 5.5.1-3299 listening on SMTP port 1025 POP port 1110 IMAP port 1143 CALDAV port 1080 LDAP port 1389 
2023-11-14 08:41:48,132 DEBUG [CheckRelease] davmail.http.HttpClientAdapter  - GET http://davmail.sourceforge.net/version.txt
2023-11-14 08:41:48,230 DEBUG [CheckRelease] davmail.http.HttpClientAdapter  - Redirect http://davmail.sourceforge.net/version.txt to https://davmail.sourceforge.net/version.txt
2023-11-14 08:41:48,241 DEBUG [CheckRelease] davmail.http.DavGatewaySSLSocketFactory  - createSocket davmail.sourceforge.net 443
2023-11-14 08:41:48,915 DEBUG [CheckRelease] davmail.DavGateway  - DavMail released version: 6.2.0-3464
2023-11-14 08:41:48,918 INFO  [CheckRelease] davmail  - A new version (6.2.0-3464) of DavMail Gateway is available !
2023-11-14 08:42:08,390 DEBUG [davmail.imap.ImapServer] davmail  - Connection from /127.0.0.1 on port 1143
2023-11-14 08:42:08,436 INFO  [davmail.imap.ImapServer] davmail.connection  - CONNECT - 127.0.0.1:51499 
2023-11-14 08:42:08,631 DEBUG [ImapConnection-51499] davmail.exchange.ExchangeSession  - Test configuration status: 401
2023-11-14 08:42:08,632 DEBUG [ImapConnection-51499] davmail  - > * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE] IMAP4rev1 DavMail 5.5.1-3299 server ready
2023-11-14 08:42:08,676 DEBUG [ImapConnection-51499] davmail  - < LOGIN ********
2023-11-14 08:42:08,751 WARN  [ImapConnection-51499] davmail.exchange.auth.O365Token  - refresh token failed javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2023-11-14 08:42:09,858 DEBUG [URL-Loader-1] davmail.exchange.auth.O365InteractiveAuthenticatorFrame  - openConnection https://login.microsoftonline.com/common/oauth2/authorize?client_id=d3590ed6-52b3-4102-aeff-aad2292ab01c&response_type=code&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_mode=query&login_hint=ifinkelstein%40cm.utexas.edu&resource=https%3A%2F%2Foutlook.office365.com
2023-11-14 08:42:09,868 DEBUG [URL-Loader-1] davmail.exchange.auth.O365InteractiveAuthenticatorFrame  - Disable integrity check on external resources at https://login.microsoftonline.com/common/oauth2/authorize?client_id=d3590ed6-52b3-4102-aeff-aad2292ab01c&response_type=code&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_mode=query&login_hint=ifinkelstein%40cm.utexas.edu&resource=https%3A%2F%2Foutlook.office365.com
2023-11-14 08:42:09,869 DEBUG [URL-Loader-1] davmail.exchange.auth.O365InteractiveAuthenticatorFrame  - Ignore Accept-Encoding
2023-11-14 08:42:09,928 DEBUG [URL-Loader-1] davmail.http.DavGatewaySSLSocketFactory  - createSocket login.microsoftonline.com 443
2023-11-14 08:42:10,417 DEBUG [URL-Loader-1] davmail.exchange.auth.HttpURLConnectionWrapper  - {null=[HTTP/1.1 302 Found], x-ms-ests-server=[2.1.16729.6 - EUS ProdSlices], X-Content-Type-Options=[nosniff], Pragma=[no-cache], P3P=[CP="DSP CUR OTPi IND OTRi ONL FIN"], Date=[Tue, 14 Nov 2023 14:42:10 GMT], Strict-Transport-Security=[max-age=31536000; includeSubDomains], Cache-Control=[no-store, no-cache], Set-Cookie=[, , , , , ], Expires=[-1], Content-Length=[838], X-XSS-Protection=[0], x-ms-request-id=[e30493e5-3c6c-4792-80f3-3ea060e49900], Location=[https://login.austin.utexas.edu/adfs/ls/?login_hint=ifinkelstein%40cm.utexas.edu&client-request-id=3e3d1714-efa3-4b2e-be3e-c0698a0fbf7f&username=ifinkelstein%40cm.utexas.edu&wa=wsignin1.0&wtrealm=urn%3afederation%3aMicrosoftOnline&wctx=estsredirect%3d2%26estsrequest%3drQQIARAA02I21DO0UkkxNrU0SE0x0zU1SjLWNTE0MNJNTE1L001MTDEysjRKTDIwTC4S4hIQEbe1W_xez3uf3YHMLv799asYJUuL8qwyU0vSrMrTrfITS0syrIz0DKzy85N2MDJeYGR8wcg4iUkqMy0zLzs1p7gkNTPPITlXr7QktSKxWC81pfQWE7-_I1CXEYjIL8qsSv3ExFpYmlpUuYpZNqOkpKDYSl8_v7QkJz8_Wy8_LS0zOdXYzFQvOT93EzMbkMzNz7vBzHiBhfEVC5MB6w8WxkWsQHc-nczyOMdmknvDZ7sFCU9mMpxi1bfw80lKDsgocavycjSqdPQxDXFP99EPyo6qSvcxCajKd4-0zIkoj_SoCrCwNbYynMDG-IGNsYOdYRcnbi8e4GX4wdey5-yEf4t_vvXYIMAAAA2#], Content-Type=[text/html; charset=utf-8]}
2023-11-14 08:42:10,421 DEBUG [URL-Loader-1] davmail.exchange.auth.HttpURLConnectionWrapper  - {null=[HTTP/1.1 302 Found], x-ms-ests-server=[2.1.16729.6 - EUS ProdSlices], X-Content-Type-Options=[nosniff], Pragma=[no-cache], P3P=[CP="DSP CUR OTPi IND OTRi ONL FIN"], Date=[Tue, 14 Nov 2023 14:42:10 GMT], Strict-Transport-Security=[max-age=31536000; includeSubDomains], Cache-Control=[no-store, no-cache], Set-Cookie=[, , , , , ], Expires=[-1], Content-Length=[838], X-XSS-Protection=[0], x-ms-request-id=[e30493e5-3c6c-4792-80f3-3ea060e49900], Location=[https://login.austin.utexas.edu/adfs/ls/?login_hint=ifinkelstein%40cm.utexas.edu&client-request-id=3e3d1714-efa3-4b2e-be3e-c0698a0fbf7f&username=ifinkelstein%40cm.utexas.edu&wa=wsignin1.0&wtrealm=urn%3afederation%3aMicrosoftOnline&wctx=estsredirect%3d2%26estsrequest%3drQQIARAA02I21DO0UkkxNrU0SE0x0zU1SjLWNTE0MNJNTE1L001MTDEysjRKTDIwTC4S4hIQEbe1W_xez3uf3YHMLv799asYJUuL8qwyU0vSrMrTrfITS0syrIz0DKzy85N2MDJeYGR8wcg4iUkqMy0zLzs1p7gkNTPPITlXr7QktSKxWC81pfQWE7-_I1CXEYjIL8qsSv3ExFpYmlpUuYpZNqOkpKDYSl8_v7QkJz8_Wy8_LS0zOdXYzFQvOT93EzMbkMzNz7vBzHiBhfEVC5MB6w8WxkWsQHc-nczyOMdmknvDZ7sFCU9mMpxi1bfw80lKDsgocavycjSqdPQxDXFP99EPyo6qSvcxCajKd4-0zIkoj_SoCrCwNbYynMDG-IGNsYOdYRcnbi8e4GX4wdey5-yEf4t_vvXYIMAAAA2#], Content-Type=[text/html; charset=utf-8]}
2023-11-14 08:42:10,423 DEBUG [URL-Loader-1] davmail.exchange.auth.O365InteractiveAuthenticatorFrame  - <html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://login.austin.utexas.edu:443/adfs/ls/?login_hint=ifinkelstein%40cm.utexas.edu&amp;client-request-id=3e3d1714-efa3-4b2e-be3e-c0698a0fbf7f&amp;username=ifinkelstein%40cm.utexas.edu&amp;wa=wsignin1.0&amp;wtrealm=urn%3afederation%3aMicrosoftOnline&amp;wctx=estsredirect%3d2%26estsrequest%3drQQIARAA02I21DO0UkkxNrU0SE0x0zU1SjLWNTE0MNJNTE1L001MTDEysjRKTDIwTC4S4hIQEbe1W_xez3uf3YHMLv799asYJUuL8qwyU0vSrMrTrfITS0syrIz0DKzy85N2MDJeYGR8wcg4iUkqMy0zLzs1p7gkNTPPITlXr7QktSKxWC81pfQWE7-_I1CXEYjIL8qsSv3ExFpYmlpUuYpZNqOkpKDYSl8_v7QkJz8_Wy8_LS0zOdXYzFQvOT93EzMbkMzNz7vBzHiBhfEVC5MB6w8WxkWsQHc-nczyOMdmknvDZ7sFCU9mMpxi1bfw80lKDsgocavycjSqdPQxDXFP99EPyo6qSvcxCajKd4-0zIkoj_SoCrCwNbYynMDG-IGNsYOdYRcnbi8e4GX4wdey5-yEf4t_vvXYIMAAAA2#">here</a>.</h2>
</body></html>

2023-11-14 08:42:10,426 DEBUG [URL-Loader-1] davmail.exchange.auth.O365InteractiveAuthenticatorFrame  - 
2023-11-14 08:42:10,427 DEBUG [URL-Loader-2] davmail.exchange.auth.O365InteractiveAuthenticatorFrame  - openConnection https://login.austin.utexas.edu/adfs/ls/?login_hint=ifinkelstein%40cm.utexas.edu&client-request-id=3e3d1714-efa3-4b2e-be3e-c0698a0fbf7f&username=ifinkelstein%40cm.utexas.edu&wa=wsignin1.0&wtrealm=urn%3afederation%3aMicrosoftOnline&wctx=estsredirect%3d2%26estsrequest%3drQQIARAA02I21DO0UkkxNrU0SE0x0zU1SjLWNTE0MNJNTE1L001MTDEysjRKTDIwTC4S4hIQEbe1W_xez3uf3YHMLv799asYJUuL8qwyU0vSrMrTrfITS0syrIz0DKzy85N2MDJeYGR8wcg4iUkqMy0zLzs1p7gkNTPPITlXr7QktSKxWC81pfQWE7-_I1CXEYjIL8qsSv3ExFpYmlpUuYpZNqOkpKDYSl8_v7QkJz8_Wy8_LS0zOdXYzFQvOT93EzMbkMzNz7vBzHiBhfEVC5MB6w8WxkWsQHc-nczyOMdmknvDZ7sFCU9mMpxi1bfw80lKDsgocavycjSqdPQxDXFP99EPyo6qSvcxCajKd4-0zIkoj_SoCrCwNbYynMDG-IGNsYOdYRcnbi8e4GX4wdey5-yEf4t_vvXYIMAAAA2#
2023-11-14 08:42:10,441 DEBUG [URL-Loader-2] davmail.http.DavGatewaySSLSocketFactory  - createSocket login.austin.utexas.edu 443
2023-11-14 08:42:10,507 DEBUG [URL-Loader-2] davmail.exchange.auth.HttpURLConnectionWrapper  - {null=[HTTP/1.1 200 OK], Server=[Microsoft-HTTPAPI/2.0], X-Content-Type-Options=[nosniff], Pragma=[no-cache], Date=[Tue, 14 Nov 2023 14:42:10 GMT], X-Frame-Options=[DENY], Strict-Transport-Security=[max-age = 31536000], Cache-Control=[no-cache,no-store], Content-Security-Policy=[default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; frame-src api-ee0b7ac0.duosecurity.com login.austin.utexas.edu www.austin.utexas.edu urn:amazon:cognito:sp:us-east-1_Ce9EEG96q urn:amazon:cognito:sp:us-east-1_xdqf1Wxuv urn:amazon:cognito:sp:us-east-1_DW1O3y2f9], Expires=[-1], X-XSS-Protection=[1; mode=block], Content-Length=[29772], Content-Type=[text/html; charset=utf-8]}

esabol commented 7 months ago

@ifinkelstein wrote

After upgrading to OSX Sonoma, DavMail crashes as soon as it enters the O365Interactive authentication frame. I cannot progress to an authentiation screen in manual mode for some reason.

You refer to "manual mode", but O365Interactive is not O365Manual. If you haven't actually tried O365Manual mode, I recommend that you try that. O365Interactive creates an interactive browser window, while O365Manual uses a simple dialog box and is more likely to work. Make sure your properties file contains the following lines:

davmail.mode=O365Manual
davmail.oauth.persistToken=true

ifinkelstein commented 7 months ago

Thank you for the suggestion.

After changing these lines in my .davmail.properties I can navigate to my institution's Duo 2FA screens. DavMail doesn't crash.

However, I get redirected to https://login.microsoftonline.com/login.srf where I am prompted "Are you trying to sign in to Microsoft Office?"

Clicking "Continue" throws up Safari cannot open the page because the address is invalid. dialog box.

I don't see a blank page with an authentication URL.

Update: I use the Outlook desktop client id. Not sure if this is part of the problem.

davmail.oauth.clientId=d3590ed6-52b3-4102-aeff-aad2292ab01c
davmail.oauth.redirectUri=urn:ietf:wg:oauth:2.0:oob

May be related to this discussion: https://sourceforge.net/p/davmail/discussion/644057/thread/a7061b73cb/

Any advice would be appreciated!

esabol commented 7 months ago

@ifinkelstein: I don't know. I have davmail.oauth.redirectUri=https://login.microsoftonline.com/common/oauth2/nativeclient and a different clientId in my .davmail.properties file.

What's your setting for davmail.url?

Your issue might be related to issue #284 if your institution is using a different endpoint than outlook.office365.com/login.microsoftonline.com.

See also issue #248 for another case of Duo MFA apparently not working with O365Manual.

ifinkelstein commented 7 months ago

I tried the davmail.url you mentioned above, but it doesn't work with the Outlook desktop client id.

The core issue, I think, is that DavMail crashes when opening an authentication window in Interactive mode in OS X Sonoma.

esabol commented 7 months ago

I tried the davmail.url you mentioned above, but it doesn't work with the Outlook desktop client id.

Huh? I didn't mention a davmail.url value. I asked you what you had for your davmail.url setting, and you didn't answer!

Did you mean davmail.oauth.redirectUri? If so, I think you are correct. Is there no other clientId you can use?

The core issue, I think, is that DavMail crashes when opening an authentication window in Interactive mode in OS X Sonoma.

And that sure seems like a problem with Java on Sonoma. Report to Oracle at https://bugreport.java.com/bugreport/ ?

If you are desperate for a more near-term workaround, I would try installing XQuartz and Docker and running DavMail in a Linux Docker container on your Mac using something like this Docker image: https://hub.docker.com/r/jberrenberg/davmail

esabol commented 7 months ago

Is there no other clientId you can use?

DavMail's default clientId is facd6cff-a294-4415-b59f-c5b01937d7bd. Try that.

davmail.oauth.clientId=facd6cff-a294-4415-b59f-c5b01937d7bd
davmail.oauth.redirectUri=https://login.microsoftonline.com/common/oauth2/nativeclient

mguessan commented 6 months ago

You may want to try to embed Azul JRE inside DavMail app, warning this is untested but similar approach is working fine on windows:

To get all features, run with latest Zulu JRE FX:

Grab latest Zulu JDK with OpenjFX: https://www.azul.com/downloads/?version=java-21-lts&os=macos&architecture=arm-64-bit&package=jre-fx#zulu
Unzip inside app package: Contents/Frameworks/zulu-jre-fx/Contents/Home

Uncomment LSEnvironment line in Info.plist:

<key>LSEnvironment</key>
<dict>
<key>JAVA_HOME</key>
<string>Contents/Frameworks/zulu-jre-fx/Contents/Home</string>
<dict>

Please let me know it this is working so I can update documentation

ifinkelstein commented 6 months ago

Thank you everyone for trying to troubleshoot this issue. I ended up reinstalling Ventura 13.6.2 as a fresh re-install. Needed to do this for other reasons.

I also installed Oracle Java 8-391 targeting ARM64.

This solved all DavMail crashes. So maybe I had a corrupt Java?

Thank you again. Marking this closed

mguessan / davmail

DavMail 6.xx and 5.5.xx crash when using O365Interactive on OSX Sonoma 14.1.1 (23B81) #316