Major isses with Thunderbird 115

[haarp]

Description

Hello!

I've been using this extension for a few months now. I eventually got it working reasonably well using POP3 mailboxes, some of which used OAuth. I was pretty happy with it, so thanks a lot for making this! :)

Since then I have upgraded Thunderbird to 115. This addon kept working, until MS Office 365 invalidated my refresh token (which is BS because it was actively being used, to fetch mail!):

mailnews.oauth: Error response from the authorization server: invalid_grant; AADSTS700082: The refresh token has expired due to inactivity. The token was issued on ... and was inactive for 90.00:00:00. Trace ID: ... Correlation ID: ... Timestamp: ...; https://login.microsoftonline.com/error?code=700082 OAuth2.jsm:331:20

Of course Thunderbird being Thunderbird, it never notified me. So I deleted the entry from KeepassXC and Thunderbird opened up the website to fetch a new one. That's when bugs started piling up. To name a few:

• Thunderbird would store the token in the internal password manager. Not even setting signon.rememberSignons=false could prevent that
• KXC-Mail would not store the token
• KXC-Mail would ask to store the token, but upon selecting yes would not
• KXC-Mail would store the token, but then ask to store it again, even if it just grabbed the same existing one from KXC
• A stored token wasn't being used, instead always bringing up the website to request a new token

So I pressed Clear storage of selected entries in this addon's settings. This "storage" seems to be a mapping between KeePass entries and mail accounts, correct? That's when all my mail accounts started breaking randomly and irreparably.

• The addon would ask which KXC entry to link again -> This is fine
• Mail accounts would sometimes use the linked credentials, sometimes not
• They'd sometimes pop up a password prompt despite credentials being stored and linked
• In cause of OAuth, they'd sometimes pop up the fetch-new-token website instead of using stored and linked tokens, sometimes not
• Restarting Thunderbird would randomize which accounts worked, and which exhibited which random error
• OAuth accounts are especially buggy. Thunderbird will pretend to fetch mail, but stall. After setting mailnews.pop3.loglevel=All:

mailnews.pop3.66: Connecting to pop://pop.googlemail.com:995 Pop3Client.jsm:141:18
mailnews.pop3.66: Connected Pop3Client.jsm:295:18
mailnews.pop3.66: S: +OK Gpop ready for requests from 123.123.123.123 v4mb136453595edb

Pop3Client.jsm:348:18
mailnews.pop3.66: C: CAPA Pop3Client.jsm:508:20
mailnews.pop3.66: S: +OK Capability list follows
USER
RESP-CODES
EXPIRE 0
LOGIN-DELAY 300
TOP
UIDL
X-GOOGLE-RICO
SASL PLAIN XOAUTH2 OAUTHBEARER
.
Pop3Client.jsm:348:18
mailnews.pop3.66: Possible auth methods: XOAUTH2 Pop3Client.jsm:597:18
mailnews.pop3.66: Current auth method: XOAUTH2 Pop3Client.jsm:668:18
mailnews.pop3.66: C: AUTH XOAUTH2 Pop3Client.jsm:508:20
mailnews.pop3.66: S: + 

and just sit there until the connection is closed, then log:

mailnews.pop3.9: NetworkTimeoutError: a Network error occurred
mailnews.pop3.9: SecurityError info:

which definitely was not a network timeout! Worse, Thunderbird typically will not notify the user when it can't fetch mail.

Expected Behaviour

• Completely disable Thunderbird's internal pw manager
• Always use linked KXC entries, if they exist
• Successfully fetch mail, even with OAuth

Current Behaviour

See above

Possible Solution

Kick Thunderbird in the butt :)

Steps to Reproduce (for bugs)

1. Ceate a fresh Thunderbird 115 profile
2. Try to add a POP3 account with regular auth, another one with OAuth (e.g. Gmail)
3. Try to fetch mail
4. Observe if mail is actually fetched or Thunderbird only pretends to
5. Observe error console, try setting mailnews.pop3.loglevel=All to get usable logs
6. Restart Thunderbird a couple of times, see if that changes things

Context

I'm unable to use Thunderbird with this extension at the moment. Every time TB starts, bugs and prompts keep popping up. I eventually created a new Thunderbird profile, hoping it would fix things. It didn't. For now, I've given up on this extension, but I'll be happy to help and test in any way I can!

I couldn't find how to get the extension's own console log, but will Show some logs from it once I know how.

Your Environment

• KeePassXC-mail version used: 1.6.1
• KeePassXC version: 2.7.6
• Thunderbird version: 115.5.0
• Operating system and version: Gentoo Linux

[haarp]

Ok, I just observed the auth stalling problem with password auth too.

Thunderbird log:

console.debug: mailnews.pop3.3: "Connecting to pop://pop.gmx.net:995"
console.debug: mailnews.pop3.3: "Connected"
console.debug: mailnews.pop3.3: "S: +OK POP server ready H migmx107 1N2SL1-1rHBt41CNr-013ggT\r\n"
console.debug: mailnews.pop3.3: "C: CAPA"
console.debug: mailnews.pop3.3: "S: +OK Capability list follows\r\nTOP\r\nUIDL\r\nUSER\r\nSASL PLAIN\r\nIMPLEMENTATION trinity\r\n.\r\n"
console.debug: mailnews.pop3.3: "Possible auth methods: USERPASS,PLAIN"
console.debug: mailnews.pop3.3: "Current auth method: USERPASS"
console.debug: mailnews.pop3.3: "C: USER a@b.c"
console.debug: mailnews.pop3.3: "S: +OK password required for user \"a@b.c\"\r\n"
... nothing happening for a while ...
console.error: mailnews.pop3.3: "NetworkTimeoutError: a Network error occurred"
console.debug: mailnews.pop3.3: "Connection closed."
console.debug: mailnews.pop3.3: "Connecting to pop://pop.gmx.net:995"
console.error: mailnews.pop3.3: "SecurityError info: "
console.debug: mailnews.pop3.3: "Done with status=2152398862"
console.debug: mailnews.pop3.3: "Connected"
console.debug: mailnews.pop3.3: "Connection closed."

Addon log (that's all):

2023-11-30 15:28:20.785: got credential request: 
Object { host: "pop3://pop.gmx.net", login: "a@b.c", loginChangeable: false, openChoiceDialog: true }
main.js:26:11

[haarp]

Hmm, I seem to be getting closer to a reproduction.

I disabled all automatic mail fetching, restarted Thunderbird, and test the accounts one-by-one.

• 2 accounts with POP3+password: work fine so far
• Gmail with POP3+OAuth: works fine so far
• O356 with POP3+OAuth: Consistently runs into these issues from the start. Both issues seem related:
  • Grabs KXC entry, immediately rewrites the same entry. I don't know why it thinks something changed (#90).
  • Stalls during authentication

Thunderbird log:

console.debug: mailnews.pop3.6: "Connecting to pop://outlook.office365.com:995"
console.debug: mailnews.pop3.6: "Connected"
console.debug: mailnews.pop3.6: "S: +OK The Microsoft Exchange POP3 service is ready. [...]\r\n"
console.debug: mailnews.pop3.6: "C: CAPA"
console.debug: mailnews.pop3.6: "S: +OK\r\nTOP\r\nUIDL\r\nSASL PLAIN XOAUTH2\r\nUSER\r\n.\r\n"
console.debug: mailnews.pop3.6: "Possible auth methods: XOAUTH2"
console.debug: mailnews.pop3.6: "Current auth method: XOAUTH2"
console.debug: mailnews.pop3.6: "C: AUTH XOAUTH2"
console.debug: mailnews.pop3.6: "S: + \r\n"
... nothing happens for 60 seconds ...
console.debug: mailnews.pop3.6: "S: -ERR Connection is closed. 12\r\n"
console.debug: mailnews.pop3.6: "Connection closed."
console.debug: mailnews.pop3.6: "Done with status=0"

KXC-Mail Log:

2023-11-30 20:04:40.401: Use saved native application de.kkapsner.keepassxc_mail main.js:26:11
KeePassXC-Mail: Connecting to native messaging host de.kkapsner.keepassxc_mail client.js:317:13
KeePassXC-Mail: Server public key: ...
2023-11-30 20:05:57.759: got credential request: Object { login: "a@b.c", host: "oauth://login.microsoftonline.com", openChoiceDialog: true } main.js:26:11
2023-11-30 20:05:57.896: keepassXC provided 1 logins main.js:26:11
2023-11-30 20:05:57.900: Use last selected entry for a@b.c@oauth://login.microsoftonline.com main.js:26:11
2023-11-30 20:05:58.483: Got new password for a@b.c at oauth://login.microsoftonline.com main.js:26:11
2023-11-30 20:05:58.626: Use last store at entry for a@b.c@oauth://login.microsoftonline.com main.js:26:11
2023-11-30 20:05:58.628: Get or create password group main.js:26:11
2023-11-30 20:05:58.819: Saving password to database for a@b.c at oauth://login.microsoftonline.com main.js:26:11
2023-11-30 20:05:58.820: Using uuid: faead8b364354cb1a3293ee1332b0932

edit:

I reset the mapping with Clear storage of selected entries. Now KXC-Mail asks me "Do you want to save the entered password for a@b.c on oauth://login.microsoftonline.com to the KeePass database?", despite the entry already existing.

• Answers Yes: Auth stall bug occurs.
• Answers No: Auth succeeds

I probably ticked "Do not ask again" at some point in the past, hence why it consistently failed. In any case, it's clear that whatever bug causes it to detect "new" credentials also causes the auth stall bug.

[kkapsner]

Sorry for the late answer... was very busy in December.

I think I see the problem... Please check if you have a confirmation dialog in KeePassXC to overwrite the existing password entry. I do not see "Saving done" in the output of your log. So the new token is sent to KeePassXC but this communication fails at some point.

Office 365 really updates the token every time - so we have to update it very often. I would recommend to have a look at the KeePassXC advanced browser integration setting "Never ask before updating credentials" (https://keepassxc.org/docs/KeePassXC_UserGuide#_advanced_usage)

[haarp]

Also sorry for the delay, just got around to testing this.

You're right! At least in this instance, the extension blocks because it cannot save the credentials. KeepassXC is not showing me any confirmation prompt, but enabling Never ask before updating credentials seems to solve it! 💪 Now I'm getting Saving done and auth succeeds.

Thunderbird needs to be restarted if it ran into an auth stall once, as it doesn't seem to recover on its own.

It's probably a good idea to add some kind of timeout here, and gracefully handle credentials not being able to be saved. It's also odd that KeepassXC does not show any dialog.

[shuther]

On top of the Never ask before updating credentials, is there a way to select a default database? I keep getting a request to select on which Keepassxc database I should save the credentials?

[kkapsner]

I did not find a setting for that. Maybe open a ticket in the keepassXC repository to request it.