Email 'preview' improvements

[momorgan]

Every time I've clicked to preview an email, I've never actually gotten to preview the email, I just see "We couldn't find any mail from that sender in your mailbox in the last 6 months."

So am I misunderstanding what this functionality should do? My expectation would be it would show me the body of the screened email, at least in part.

[Jivings]

We're getting multiple reports of this but I can't recreate it on any of my accounts.

What type of account is it? (gmail, outlook etc?). The only thing I can think of is that maybe we're not searching the correct folders, or the type of search I'm using isn't support by the mail client.

[momorgan]

Ooh interesting. I've just checked and I'm getting the same results across three accounts - one at gmail, one at zoho and one at runbox.

[Jivings]

Can you run a search for the same email address in Gmail and see what gets returned? If you could let me know what labels it has for example that would be super useful.

[momorgan]

Okay so I got a really interesting outcome.

I've had the same email basically twice (which as it turns out is kinda fortunate), the from address is in a rollup. One is in _lma-rollups, the other in _lma-rollups/screened

Back in the interface:

Both listed there. Click preview on the first one, and preview works! Never seen that before. Click preview on the second, it throws the error.

[Jivings]

Okay I think I've cracked it.

For non-gmail accounts we were accidentally only searching the first folder (whatever that might be). It should search all the folders now.

There's a problem with this where it might take a long time for non-gmail mailboxes, but we'll see how much of a pain that is. Going forward I might be able to think of a way to mitigate it a bit.

[momorgan]

Cool. Does that address the above example, though? I picked an example from a Gmail account like you asked

[Jivings]

Can you try now and see if it still fails?

[momorgan]

Hm, so I tried the above again. Top one works, bottom one shows a new error:

[Jivings]

Can't see any evidence of this problem now 😅 can you try again with some others in case the browser is caching the response for that one or something?

[Jivings]

Crap, I just found another issue with it that could cause this but I've shipped a quick fix. Please retry 😅

Thanks for your patience 🙏

[momorgan]

Well look at that, I think you've done it ;)

The example above now works too -- the second preview loads very slowly but it does load.

[Jivings]

Phew 😅

Thanks for sticking with it! I appreciate it :)

[momorgan]

Ah sorry, I don't think we're quite out of the woods on this one yet.

So I have an email in the screened list, and correctly sitting in '_lma-shield/screened'. When I click to preview, one of two things happens. Either I quickly get "Something went wrong fetching your mail preview" or I get "Loading mail preview, please wait..." and it sits like that for ages. Longer than I've managed to wait, anyway.

[Jivings]

For every item? If there's a specific email provider it doesn't work for then can you let me know what it is?

[momorgan]

This is on runbox & zoho. Seems fine on gmail. I've tried about 20 items in all.

[momorgan]

Small update: I think it's more intermittent than I made out, as I have found emails on the same servers/folders that preview fine.

I wonder if it might help for the error to return an item-specific ID so I could point you to the ones that work and the ones that fail?

[Jivings]

I've just shipped a big change for this so hopefully it's working much better now. I think the problem was within our queuing system which is why is didn't make any sense to me initially.

Try now! :)

[momorgan]

I think I'd have to say my try-outs are bit inconclusive... I've not yet found an item that throws an error: everything I've tried shows "Loading mail preview, please wait...".

I think the problem now is the time: some seem return within seconds, but most do not. I thought perhaps it still wasn't working but maybe it is, just very slowly. For a sense of scale, I timed one: it took 1min 34sec. From here I can't yet see what would cause some to be faster than others.

[Jivings]

Due to how IMAP works we have to search one mailbox folder at a time, and opening the next folder to search takes a second or two.

There's no universal way to say "this folder is more important than another", so we have an algorithm that tries to order them in a sensible way to make the searching faster (eg we always search INBOX first if it exists). Sometimes the item we're searching for is found quickly, and sometimes not depending on what folder it was in.

The actual search part is usually very fast, it's the box opening that's slow.

Gmail has a folder called "All Mail", which contains everything. So we only need to open that one folder to do any search - which is why Gmail is so much faster.

There are three improvements we could make that I can think of in order of usefulness:

1. When we do the scan for subscriptions, we store the folder that subscription was in. Then when we query for the preview, we search that folder first. It wont always have stayed in the same place, but it probably will be there.
2. When the preview button is clicked we intelligently choose which folder to search first. Eg if the mail was rolled up then it's probably in _lma-rollups, if it's blocked then probably it's in _lma-screened.
3. When we move mail (for the screener or rollups) we store the place we moved it to and we search that folder first.

[momorgan]

Yes, I'd agree with all of that. I could suggest a couple more from a users' perspective:

1. I've been testing preview buttons to death, but this isn't representative of typical user behaviour. Realistically, where they're most useful is when emails first arrive: as part of the keep/block/unsubscribe decision process. If I wanted to see an email LMA had handled more than a week ago, it'd be much easier for me to track it down in the mail client (or look at mail history). So maybe this whole thing can be made easier by only putting preview buttons on recently-received emails (a week or so?). As you say, these are the ones most likely to be wherever LMA put them.

2. Some more tangible feedback in the UI of what the delay might be would help manage users' expectation. There's a number of forms this could take, from simple wording to some kind of more accurate calculation.

3. A minor observation but I'll capture it anyway: So that I could manage the preview delay and also keep working in LMA, I found myself subconsciously centre-clicking to open previews in a new tab, which opens to /app/mail/screener/preview and throws an error. I wonder if that presents an opportunity in some way.

hth!

[Jivings]

Okay here's what we'll do:

• [ ] return original mailbox information with subscription mail when it's scanned
• [ ] provide mailbox hints to the scanner so we can specify which folders to search first
• [ ] create deep links for the preview so it can be opened in a new tab

[Jivings]

This improvement is being rolled into the new scanner changes for the Gmail API 👍

https://github.com/leavemealone-app/roadmap/issues/51

[Jivings]

This is done 👍