search

Extended-Thunder / send-later

Schedule messages to be sent at a specific time.
Mozilla Public License 2.0
124 stars 17 forks source link

Mail sent using "Send later" never leaves Thunderbird #16

Closed padelt closed 3 years ago

padelt commented 4 years ago

Pinned note from @jperryhouts about the current status of this issue:

At long last it looks like we have a promising solution for the bug, which is available in the latest release version (8.4.4 as of this comment).

If you are experiencing this bug, please give that release version a try and post here if you are still having problems. I hope to have this version available via addons.thunderbird.net as soon as possible.

Running on Thunderbird 78.2.2 (32-Bit). When I compose a message, I have the "Später Senden" (Sent Later) button on the top right. Clicking it show the expected UI: The mail is saved to the drafts folder as expected. I cannot find the column for the "send time" in the Drafts folder of two different mail servers (a hosted one by Hosteurope and on self-hosted Dovecot): When I look in the headers of the draft mail, I see this (redacted some values):

FCC: imap://user%40domain.name.net@domain.name.net/Sent
X-Identity-Key: id2
X-Account-Key: account3
BCC: my-default-bcc@domain.name.net
From: User Name <info@domain.name.net>
Subject: My Subject
To: Destination User <user@destination.com>
References: <XXX@destination.com>
Message-ID: <YYY@domain.name.net>
X-Send-Later-At: 2020-09-24T09:51:00+02:00
X-Send-Later-Recur: none
X-Enigmail-Draft-Status: N00200
Date: Thu, 24 Sep 2020 09:50:27 +0200
X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0;
 attachmentreminder=0; deliveryformat=4
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.2.2
MIME-Version: 1.0
In-Reply-To: <XXX@destination.com>
Content-Type: text/html; charset=utf-8
Content-Language: de-DE
Content-Transfer-Encoding: 8bit

Problem is: The mail never leaves Thunderbird. It just sits there. "Never" means: I have let it sit for 12-24h and it is still there.

The error console (ctrl+shift+j): further down:

I tried disabling Enigmail and restarting Thunderbird. Also tried disabling OpenPGP for the accounts. Neither made a difference.

Is there anything I can do to help diagnosing/debugging/fixing this?

jperryhouts commented 3 years ago

@jobisoft I noticed that your brute force extension doesn't quite replicate the format that Send Later uses, because the callback function in your example is a regular function, not an async one.

jperryhouts commented 3 years ago

@jperryhouts I think it would be helpful for you to add an update to the beginning of the description of this issue (i.e., all the way at the top of this page, where people will see it first), explaining the current status and specifically what people can do to help debug it, in terms of providing information here. There are a lot of comments here so it's difficult to find that information buried in them.

Good call. Unfortunately github doesn't let you pin messages, but I edited the original bug report with some information.

jperryhouts commented 3 years ago

I changed the order in which send later attempts to fetch messages to more closely align with the way @jobisoft did it in his brute force test extension. This shouldn't make a difference, but I've experienced other cases where async/await causes a function to somehow lose its execution context, which would be consistent with everything we've seen so far about this bug.

If anyone would like to give that a try, here's a package of the current main branch state.

send_later-8.4.0.zip

jperryhouts commented 3 years ago

The more I think about that possibility the more it makes sense as a possible culprit. Send Later has always used a wrapper function that iterates over draft messages, executing a callback on each one.

Conceptually, that looks like:

async forAllDrafts(callback) {
    let drafts = await ...
    for (let msg of drafts) {
        await callback(msg);
    }
}

The way await operates is weird, and if I understand it correctly it tries to restore all state variables in its execution context when it "returns". I assume that means it generates a new anonymous function behind the scenes, and attempts to bind it to the parent context.

That should be fine but it seems precarious, and I can imagine a number of ways that could go wrong. For example, if Thunderbird fails to restore variables like 'browser' it could cause the TypeError: browser is null messages that @padelt was seeing.

I can't eliminate all of the asynchronousness, but I put together a version that uses chains of .then() instead of await in that loop, which makes possiblySendMessage a regular function. If someone who's been experiencing this bug would give this version a try it would be really helpful.

Edit: note that the version number is identical to the one I posted above, but this is a different package. send_later-8.4.0.zip

jperryhouts commented 3 years ago

Oops. I did not mean to close this, not sure how that happened.

jperryhouts commented 3 years ago

I've gone through the logic for checking draft messages again and made every effort to ensure that the execution context does not get lost or changed. If that really was the problem and this is not a deeper bug, then it might be fixed by #151.

This package is up to date with the current main branch: send_later-8.4.2.zip
send_later-8.4.1.zip

glilienthal commented 3 years ago

@jperryhouts Great! I'd love to try. How do I install this zip? (or is there an xpi file?)

padelt commented 3 years ago

@glilienthal an XPI file is just a renamed ZIP. Save it to a folder and then go to the Add-on Manager in Thunderbird, open the menu at the top right (gear icon) and select "Install add-on from File". Point to the ZIP file. Thunderbird then installs the add-on.

glilienthal commented 3 years ago

Unfortunately, this (8.4.2) doesn't work either.

I noticed something: When I used send later for the first time, it told me "I cannot save to draft folder due to network problems" (not exact quote but similar). I clicked on "retry" and it saved it in draft folder, but then never send it.

It did, however, upload it to the IMAP folder online. Here some stuff from the header which looks good: X-Send-Later-Uuid: {c9074cb8-346e-4716-8eee-48de89aa8662} X-Send-Later-At: Tue, 24 Nov 2020 12:44:00 +0100 X-Send-Later-Recur: none Message-ID: a8506254-60c3-dbcf-631d-696694592ea4@investigate-europe.eu Organization: Investigate Europe Date: Tue, 24 Nov 2020 12:38:20 +0100 X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0; attachmentreminder=0; deliveryformat=4 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0

Thanks @padelt for answering my nooby question.

glilienthal commented 3 years ago

and here the logfile as suggested in the OP (deleted)

padelt commented 3 years ago

Random details I noticed:

  1. We both seem to be from Germany. Maybe it has something to do with localization. (I should mention that I am running an english version of Thunderbird now.)
  2. I always wondered if this error is related: [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]" nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)" location: "JS frame :: resource://gre/modules/L10nRegistry.jsm :: L10nRegistry.loadSync :: line 658" data: no] I see that in your debug log too but don't know if this is "normal" as in "every Thunderbird installation logs that".
  3. You are running 32bit Thundbird, I am using 64bit Thunderbird.
jperryhouts commented 3 years ago

Unfortunately, this (8.4.2) doesn't work either.

Sigh. Back to the drawing board I guess. This is driving me crazy.

We both seem to be from Germany. Maybe it has something to do with localization. (I should mention that I am running an english version of Thunderbird now.)

I think that's a coincidence in this case. I've gotten some reports via email of the same behavior from people in the US and other European countries.

I always wondered if this error is related: [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]" nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)" location: "JS frame :: resource://gre/modules/L10nRegistry.jsm :: L10nRegistry.loadSync :: line 658" data: no]

This happens without send later installed, so I guess it's just a Thunderbird quirk. It's something related to locales and translations.

You are running 32bit Thundbird, I am using 64bit Thunderbird.

This happened to you on 32 bit as well though, right?

jperryhouts commented 3 years ago

All I can say is that I'm incredibly frustrated with this situation. The attached package current beta release (8.4.4) is my latest stab at this. In that update Send Later bundles its own alternative to the getRaw function, and reduces its reliance on unbound callback functions. This might actually fix the problem (fingers crossed), but at the very least it should generate much more helpful debugging output.

In an interesting development I actually experienced this bug myself a couple times today, but wasn't able to reliably reproduce it. Encouragingly, it has not shown up once since I implemented this patch, but it feels pretty superstitious to read much into that.

I would really appreciate it if someone who's experiencing this bug would give it a try.

pat-vu commented 3 years ago

Yes!! I tried it with Thunderbid 78.5.0 (32 bit) on windows 10, with an imap outlook account and it worked immediately, for new messages as well as all previous attempts that were left in the drafts!

A great news, MANY thanks!!

p

On 25/11/2020 01:02, Jonathan Perry-Houts wrote:

I have done almost nothing but work on this today, and all I can say is that I'm incredibly frustrated with this situation. The attached package is my best attempt to work around this, by rolling my own alternative to the getRaw function. At best this might actually fix the problem, at worst it will give us much more helpful debugging output. I would really appreciate it if someone who's experiencing this bug would give it a try. send_later-8.4.3.zip https://github.com/Extended-Thunder/send-later/files/5593466/send_later-8.4.3.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Extended-Thunder/send-later/issues/16#issuecomment-733324760, or unsubscribe https://github.com/notifications/unsubscribe-auth/ART23EFSERH7WU6GM2T2MTDSRRCSDANCNFSM4RX7RTJQ.

pat-vu commented 3 years ago

PS

But the only strange thing is that the "SendLater" clock button in the message window disappeared (not essential to me and it does work with the send later command in menu)

Thanks again

On 25/11/2020 01:02, Jonathan Perry-Houts wrote:

I have done almost nothing but work on this today, and all I can say is that I'm incredibly frustrated with this situation. The attached package is my best attempt to work around this, by rolling my own alternative to the getRaw function. At best this might actually fix the problem, at worst it will give us much more helpful debugging output. I would really appreciate it if someone who's experiencing this bug would give it a try. send_later-8.4.3.zip https://github.com/Extended-Thunder/send-later/files/5593466/send_later-8.4.3.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Extended-Thunder/send-later/issues/16#issuecomment-733324760, or unsubscribe https://github.com/notifications/unsubscribe-auth/ART23EFSERH7WU6GM2T2MTDSRRCSDANCNFSM4RX7RTJQ.

glilienthal commented 3 years ago

Great stuff! I can confirm it also works for me. (win10, TB 32Bit). I also see the clock button.

Thank you so much!

jperryhouts commented 3 years ago

Oh thank god! I was thinking last night that if this doesn't fix it I'm just throwing my computer in a river and moving to the woods.

@pat-vu Are you referring to when you're viewing an already saved message? Interesting that showed up before but not now. That also relis on the old method of reading messages, which I'll need to update to the new approach.

Anyhow this is a major relief. I'll get to ironing out details with patch over the weekend but for now keep me updated if things stop working again. I'm going to leave this open until the fixed version is on ATN, so I'll have somewhere to point users who contact me about it.

BramMeehan commented 3 years ago

Yes! That 8.4.4 worked on 78.5.0 on a Mac running Catalina.

pat-vu commented 3 years ago

@pat-vu https://github.com/pat-vu Are you referring to when you're viewing an already saved message? Interesting that showed up before but not now. That also relis on the old method of reading messages, which I'll need to update to the new approach.

Not sure to understand your question :) I think I wrote that after installing the new version I saw 1) newly created messages to be sent later were correctly sent when expected, but even older messages that were created with the previous beta versions and had not left the drafts for weeks now also were sent at the same time - so double success. And 2) the clock-shape send later button (with greenish part as logo for the extension) had disappeared but now I see it had just moved to other extremity of the command tool bar and I could put it back in place! :)

So all seems fine, thanks again a lot, and please don't go to the woods :)

PatEinstein commented 3 years ago

Pinned note from @jperryhouts about the current status of this issue:

When I compose a message, I have the "Später Senden" (Sent Later) button on the top right. Clicking it show the expected UI: The mail is saved to the drafts folder as expected. I cannot find the column for the "send time" in the Drafts folder of two different mail servers (a hosted one by Hosteurope and on self-hosted Dovecot):



Problem is: The mail never leaves Thunderbird. It just sits there. "Never" means: I have let it sit for 12-24h and it is still there.

I tried disabling Enigmail and restarting Thunderbird. Also tried disabling OpenPGP for the accounts. Neither made a difference.

When I click in the drafts folder on a message that has not being send by Send Later and I click then on the Send Later button to the right it drops a little windows with this message in red:

ERROR Message is encrypted and will not be processed by Send Later.

In the Tools menu I have set to NOT encrypt any message.

Also the column in the Drafts folder jumps always to the beginning even when I move it to somewhere else.

jperryhouts commented 3 years ago

When I click in the drafts folder on a message that has not being send by Send Later and I click then on the Send Later button to the right it drops a little windows with this message in red: ERROR Message is encrypted and will not be processed by Send Later.

In the Tools menu I have set to NOT encrypt any message.

Also the column in the Drafts folder jumps always to the beginning even when I move it to somewhere else.

These are totally different bugs.

PatEinstein commented 3 years ago

When I click in the drafts folder on a message that has not being send by Send Later and I click then on the Send Later button to the right it drops a little windows with this message in red: ERROR Message is encrypted and will not be processed by Send Later. In the Tools menu I have set to NOT encrypt any message. Also the column in the Drafts folder jumps always to the beginning even when I move it to somewhere else.

These are totally different bugs.

Thanks for the reply. I understand that the column thing is a different issue. I just wanted to contribute here and trying to help why Send Later did not send it's messages.

jperryhouts commented 3 years ago

Thank you, but I think we have this particular bug sorted out now. The other two are unrelated, but will hopefully go away in the latest version as well. For the column in particular I recommend restarting Thunderbird any time you update Send Later.

In any case, check back in when version 8.4 is available if you are still having these issues.

Nerd2k commented 3 years ago

Sorry to say, but I still experience the bug. Addon version 8.4.5 via regular addon update, no manual install. (I restarted Thunderbird several times since the update.) Due emails are still sitting indefinitely in the Drafts folder.

jperryhouts commented 3 years ago

Also, it would be really helpful if you could generate a debug log of this. To do that:

  1. Set the Send Later option "console log level" to "All", and "check every" to "1" minute;
  2. Open the log console (Ctrl+shift+J (cmd+shift+j on mac))
  3. Leave the console open while you do all of the steps to reproduce this behavior. i.e.:
    • Open a composition window
    • Prepare a test message (to yourself, presumably)
    • Schedule it to send, using something like "in 1 minute"
    • Wait for Send Later to go through at least two main loop cycles after the time at which the message should have sent.
  4. Save that debug log, and upload it here as a text attachment (please don't copy-paste it into a comment).
Nerd2k commented 3 years ago

Thanks for your reply. I believe we can narrow down the problem quite quickly. I tried to schedule a message from my other account and there send later works fine. So the problem seems to be account specific. I investigated the debug log and it shows an indication for that assumption, since for the problematic account it ouptuts a message like that (note that I replaced the account name by "XXX"):

18:08:47.983 DEBUG [SendLater]: Unable to find drafts folder for account Object { id: "account1", name: "XXX", type: "imap", folders: (9) […], identities: (1) […] }

So I suppose it has something to do with the virtual folder thing you mentioned? How can we fix that? Weirdly enough, the message is stored and scheduled correctly in the drafts folder though, i.e showing the correct scheduled date in the "Send Later" column. It's weird that it complains that it can't find the drafts folder when at the same time it is able to put the draft in there, isn't it?

jperryhouts commented 3 years ago

Huh. That's annoying, but definitely more manageable than the original problem. Is that error preceded by anything that seems relevant? That means that Send Later looped over all the folders in that account, and wasn't able to find one that it recognized as the drafts folder.

Unintuitively, there is totally separate logic for finding the drafts folder during the message saving operation, and yet another in the code related to the schedule column, so it's not entirely surprising that one could break and the others still work.

Try running the following code snippet in your console (via the >> input box at the bottom):

function listFolders(f) { console.log(`  ${f.type}: ${f.path}`); f.subFolders.map(listFolders); }; var ext = window.ExtensionParent.GlobalManager.extensionMap.get("sendlater3@kamens.us"); var apiCan = [...ext.views][0].apiCan; apiCan.findAPIPath("accounts").list().then(accts => accts.map(a => { console.log(a.name); a.folders.map(listFolders); }));

That should output something like the following:

Local Folders debugger eval code:1:311
  drafts: /Drafts debugger eval code:1:35
  trash: /Trash debugger eval code:1:35
  outbox: /Unsent Messages debugger eval code:1:35
jph@extended-thunder.org debugger eval code:1:311
  inbox: /INBOX debugger eval code:1:35
  undefined: /[Gmail] debugger eval code:1:35
  sent: /[Gmail]/Sent Mail debugger eval code:1:35
  archives: /[Gmail]/All Mail debugger eval code:1:35
  junk: /[Gmail]/Spam debugger eval code:1:35
  undefined: /[Gmail]/Starred debugger eval code:1:35
  undefined: /[Gmail]/Important debugger eval code:1:35
  drafts: /[Gmail]/Drafts debugger eval code:1:35
  trash: /[Gmail]/Trash debugger eval code:1:35
  drafts: /Drafts debugger eval code:1:35
  undefined: /Trash

Specifically, it prints the name of each account and then loops over all its subfolders, printing each filder's "type" and name. In theory each account should have at least one folder whose type is "drafts". Let me know what happens in your case, because it sounds like send later is not finding that.

jperryhouts commented 3 years ago

Since this is now about a different problem, I've opened a new issue to continue the discussion: #165

That has the added benefit of finally getting to close the infamous issue #16 for good!

padelt commented 3 years ago

Thank you @jperryhouts for all the work! I can imagine how frustrating that is/was. My version of “going to the woods” is “I’ll become a gardener”. Please don’t - either one! ;-) I really appreciate the work and SendLater!