search

joeyates / imap-backup

Backup and Migrate IMAP Email Accounts
MIT License
1.33k stars 74 forks source link

Backup marks message in mailbox as read #118

Closed GadgetComa closed 4 months ago

GadgetComa commented 2 years ago

Sorry if I'm missing something here. I searched a lot to see if I overlooked the answer..

imap-backup works well for me (thank you!) except all unread messages in my mailbox get marked as read when the backup runs. Is this expected or is there a config option I'm missing? I'm backing up two iCloud mail accounts.

Thanks!

joeyates commented 2 years ago

Hi @GadgetComa

That's odd.

When imap-backup, unread messages should remain flagged as unread. That's certainly the case for the GMail and the other providers I use.

Please re-check that you can replicate this with iCloud. If so, I will activate an iCloud account and see if I can replicate myself.

GadgetComa commented 2 years ago

Thanks for the reply. I see this happen every day on two different iCloud accounts. Is there some debug information I can provide to assist in troubleshooting? Also, if it helps, I can set up a gmail account to test and see if it’s specific to iCloud or something in my environment.

GadgetComa commented 2 years ago

It does appear to be iCloud-specific. I just did a backup of a Gmail account using the exact same setup I use for iCloud and the messages are left as unread.

joeyates commented 2 years ago

I have now reproduced this on an iCloud account. As this behaviour is limited to iCloud, and the calls made by imap-backup should not change the status of the 'SEEN' flag, I think we can consider this an iCloud bug.

darkfence commented 2 years ago

@joeyates I have encountered this today on regular IMAP account (hosted with purelymail.com). It's not a iCloud bug anymore. Any chance of looking into it?

joeyates commented 2 years ago

Hi @darkfence thanks for the extra info.

Could you run a quick check for me? Please mark some emails as unread and then re-run imap-backup. Do the emails get marked as read immediately?

darkfence commented 2 years ago

Yes, as soon as the backup starts, unread messages are marked as read.

joeyates commented 2 years ago

I've now reproduced this behaviour on an iCloud account. Next step is to create a purelymail.com to reproduce it there too. GMail (and most other providers) do not behave this way.

The "philosophical" question, I suppose, is whether IMAP servers should mark emails as read when their content is accessed in any way (as iCloud seems to do), or whether they should expect clients to set the 'Seen' flag explicitly.

I'm looking at a workaround that checks the 'Seen' flag status before and after fetching emails, and removes the 'Seen' flag if it gets set by the operation.

joeyates commented 2 years ago

@GadgetComa I've created a branch with a fix for iCloud: feature/debug-emails-being-marked-as-read

If you have a change to try it out, that would be very helpful.

You can install it as follows:

gem uninstall imap-backup # To uninstall the current version
git clone https://github.com/joeyates/imap-backup.git
cd imap-backup/
git checkout --track -b feature/debug-emails-being-marked-as-read origin/feature/debug-emails-being-marked-as-read
gem build imap-backup.gemspec
sudo gem install imap-backup-*.gem

@darkfence As soon as I've had a chance to try this out with purelymail.com, I'll let you know!

joeyates commented 2 years ago

Hi @darkfence

I've added the workaround for purelymail.com clients too. If you have a moment please try it out (see comment above).

GadgetComa commented 2 years ago

@GadgetComa I've created a branch with a fix for iCloud: feature/debug-emails-being-marked-as-read

If you have a change to try it out, that would be very helpful.

Thanks. I'll try it out and let you know.

GadgetComa commented 2 years ago

I just ran the backup on two different iCloud accounts and the unread flag is intact as expected. Thanks for the fix.

darkfence commented 2 years ago

@joeyates unfortunately it does not work on my purelymail account. Unread mail was marked as read as previously.

joeyates commented 2 years ago

@joeyates unfortunately it does not work on my purelymail account. Unread mail was marked as read as previously.

@darkfence Which domain are you using for your purelymail account?

darkfence commented 2 years ago

My own domain. Maybe that's the reason...

joeyates commented 2 years ago

OK, I'll need to add a configuration option to handle this case.

joeyates commented 1 year ago

I've released the fix as version 6.1.0

@darkfence for custom domains, you need to set the relevant flag via setup.

bdillahu commented 4 months ago

Could I suggest (unless its dangerous or something) that this option default to "on"? I've not marked all my email as "read" for all of history, even though it wasn't... My bad for not seeing the switch (or catching the problem in my testing), but just thought having the option be defaulted to "safe" might be better.

joeyates commented 4 months ago

Hi @bdillahu

I understand why defaulting this option to "on" would work well for some cases. But, as it adds a couple of round trips to each batch of downloads, it slows things down. I really believe this behaviour is "wrong", so I am loathe to apply a workaround to every account to cover a problem caused by a number of misconfigured servers (which happen to include all of Apple's!)

BTW: what email provider are you using?

As possible mitigations, I'm considering a few options:

  1. Have a whitelist of servers that don't have this behaviour. Set "off" for them, and "on" for the rest,
  2. Create a setup "wizard" that asks which setup to use,
  3. Have imap-backup run a test and see if the server is marking downloaded emails as read.

Number 3 is the most work, but I think would be nicest. If some kind soul sponsors it, I can look into it. Otherwise, PRs are always welcome!

bdillahu commented 4 months ago

Hi @bdillahu

I understand why defaulting this option to "on" would work well for some cases. But, as it adds a couple of round trips to each batch of downloads, it slows things down. I really believe this behaviour is "wrong", so I am loathe to apply a workaround to every account to cover a problem caused by a number of misconfigured servers (which happen to include all of Apple's!)

Totally understand.. makes sense.

BTW: what email provider are you using?

I'm also on purelymail - The guy that runs that tends to be very responsive - I'll drop him a note and see if its something he's aware of/could fix on his end (at least remove one from the list :-)).

As possible mitigations, I'm considering a few options:

  1. Have a whitelist of servers that don't have this behaviour. Set "off" for them, and "on" for the rest,

So far we've seen Apple and purelymail.com I believe - of course many folks with providers like purelymail use custom domains and depending on configuration, that might be hard to determine.

  1. Create a setup "wizard" that asks which setup to use,

That would be good.

  1. Have imap-backup run a test and see if the server is marking downloaded emails as read.

As you say, that would be best probably, but a pain for you :-)

Probably not on my immediate list to try to take a stab at it, but if I get free cycles, I'll take a look.

Bruce

Number 3 is the most work, but I think would be nicest. If some kind soul sponsors it, I can look into it. Otherwise, PRs are always welcome!

joeyates commented 4 months ago

@bdillahu Strange you didn't get this setting automatically.

The default for purelymail is to set the flag to "on".

bdillahu commented 4 months ago

Dunno...

By chance do you look for "imap.purelymail.com" to key off of?

It may have been self inflicted as when I initially set things up I mistyped the "imap" part - then I believe I just edited the config file directly when I realized I had a typo.

joeyates commented 4 months ago

That would explain the course of events.

It's hard to imagine how to handle this use case in a system where the config can be manually modified.

Thanks for the clarification! I'll bear this in mind in future and try to find a way to catch this type of problem.

bdillahu commented 4 months ago

The only suggestion I can come up with is to maybe check for *purelymail* to handle variations…

or have the check occur each time you read the config, but then people who particularly want it the other way would have problems, so you probably have the best solution already.