"From:" header is changed to "-From:"

LMRW commented 8 months ago

"From:" header is changed to "-From:" in MBOX backups

Is there any reason for this and can it please be disabled?

Thank you for the great project

joeyates commented 8 months ago

Hi @LMRW

I'm surprised you're seeing "-From:" replacing "From:" in MBOX files.

imap-backup does something similar because it stores messages on disk in the MBOXRD format. It replaces From at the start of lines with >From. It does the same with >From, or From preceded by any number of > characters.

The best reference for the MBOXRD format is this: http://www.digitalpreservation.gov/formats/fdd/fdd000385.shtml

A few questions:

Are you seeing this - character in all messages? Is it only at the start of lines? Are you seeing >From at all?

LMRW commented 8 months ago

I am seeing '-From:' on all messages.... I do not see '>From'.

This is via docker image theohbrothers/docker-imap-backup:latest on Arm64 Ubuntu connecting to a self hosted mail server

All other headers such as subject and to are uneffected, only From has this issue

It means when the MBOX is imported into a mail client, all emails show as from unknown sender

Below is an example:

Date: Thu, 13 Apr 2017 01:02:36 +0000
-From: Example Support <support@example.com>
Reply-To: "Example Support" <support@example.com>

I have tested a few accounts, mailboxes and clients and they all see the above structure

This is a fantastic project so i'll gladly help track this bug down if I can help!

Thank you

joeyates commented 8 months ago

Hi again @LMRW

This issue has not come up before through direct use of imap-backup.

I suspect this may be an issue related to the Docker image. If so, it should be reported here.

In order to exclude other causes, could you try to run fresh backups without using the Docker image. You would need to install imap-backup directly (see the instructions in the README).

LMRW commented 8 months ago

I have now installed without docker, following your usual instructions

The issue remains

This is Arm64 Ubuntu 22.04 LTS with real Ruby installed (not via snap)

The exact same issue remains unfortunately, ruling out Docker

Please let me know if I can help test in any other ways

LMRW commented 8 months ago

And thank you again for this project, I hope I am able to help.

ATM this bug is a bit strange and certainly an annoyance, but I would not say it is critical for my personal needs. The 'from' header is still there - and can be manually accessed. It is simply a bug in that it does not show in Apple Mail/Thunderbird etc when opening the MBOX files as those clients do not know how to handle the "-From:" header and expect "From:"

joeyates commented 8 months ago

Hi @LMRW

The best approach I can think of is to alter the imap-backup source code and see what changes.

gem info imap-backup

should tell you where it is installed. (If you have more than one version, I suggest deleting the older one(s) and re-running the backup just to check)

The actual files will be under a subdirectory, like gems/imap-backup-14.2.0.

If you open the file lib/imap/backup/email/mboxrd/message.rb, you'll see the add_extra_quote method. It's there that the extra > is added before From at the beginning of lines.

You could try the following:

Check your copy has exactly body.gsub(/\n(>*From)/, "\n>\\1")
Replace that with just body, and re-run backups. Do you get plain 'From'?
Instead of "\n>\\1", put "\n%%\\1" and see if backups come out with %% before From.

If you get a chance to try this out, let me know how it goes

LMRW commented 8 months ago

Thank you I will try and find some time to try this and get back to you

LMRW commented 8 months ago

Apologies for the delay I still intend to test this for you

dieterdanger commented 6 months ago

Hey, first of all thank you very much for this wonderful program! I have the same problem on MacOS with imap-backup installed via brew. Is there any chance for a fix in the future?

Best regards Dennis

joeyates commented 6 months ago

Hi @dieterdanger

Please run the following commands and paste the output:

sw_version

brew --version

imap-backup version

Also

Are you backing up emails from more than one account?
...from more than one provider? Which?
Does this problem occur with all accounts/providers?
If you move the backup files out of the way and run the backup again, do you still see -From?
What program are you using to view the contents of .mbox files?
Do you see -From if you use imap-backup local show EMAIL_ADDRESS FOLDER UID ?

joeyates commented 3 months ago

@LMRW @dieterdanger Any updates on this?

LMRW commented 2 months ago

I am not using MacOS I am using Ubuntu Arm64

I have the exact same issue still

Its not a huge deal, as this is a backup and its repairable in an emergency with some kind of mass file manipulation... but it is quite annoying and i'd love to help you fix it!

Interesting, I have a brand new mailserver now... 100% change, still same -from: issue

LMRW commented 2 months ago

imap-backup 14.6.1

LMRW commented 2 months ago

ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [aarch64-linux-gnu]

LMRW commented 2 months ago

And your questions...

Are you backing up emails from more than one account? -- YES

...from more than one provider? Which? -- I can PM this

Does this problem occur with all accounts/providers? -- YES, ive not found any that it doesn't

If you move the backup files out of the way and run the backup again, do you still see -From? -- YES

What program are you using to view the contents of .mbox files? -- Both Apple Mail.app & MBOX viewer (https://apps.apple.com/gb/app/fast-mbox-editor/id1444529635?mt=12) -- exact same issue on both

Do you see -From if you use imap-backup local show EMAIL_ADDRESS FOLDER UID -- Where do I find the UID please and I will test

LMRW commented 2 months ago

I found how to track UID by using your excellant -h commands, thank you -- I really do adore imap-backup.

I can confirm it shows "From:" -- correct! When I run imap-backup local show EMAIL_ADDRESS FOLDER UID

Which leads to 2 possibilities perhaps?

1) MacOS cannot read the MBOX correctly (in neither stock Apple Mail app or 3rd Party mail apps) 2) imap-backup can read its own MBOX correctly but is reading back correctly what it wrote wrong

I'll try another MBOX client if you can suggest one? I have access to Windows, Ubuntu and Mac.

At the moment, i'm afraid the mystery has only deepend.

joeyates commented 2 months ago

@LMRW Thanks for the info.

So, just to confirm what's on the disk, please open an .mbox file with a text editor , e.g. TextEdit on macOS, and check whether it has - From

As for a client, I suggest installing Thunderbird and exporting from imap-backup to Thunderbird with

imap-backup utils export-to-thunderbird EMAIL

LMRW commented 1 month ago

Multiple types when open MBOX in text editor, all below in one MBOX across various emails

From example@example.com Fri Apr 19 01:54:43 2024
From: Name <example@example.com>

From example@example.com Fri Apr 19 04:44:46 2024
Return-Path: <example@example.com>
...
>From: example@example.com

joeyates commented 1 month ago

Thanks @LMRW for the further information.

So, it looks like there is no imap-backup bug here, it's an oddity of the client used to open the messages.

LMRW commented 1 month ago

But why does exports and imports from other mail backup tools (Apple Mail, thunderbird, horcrux) not have the issue? I can export from them and open in same tools and not experience the issue

joeyates commented 1 month ago

It's certainly weird, but, it seems that imap-backup is saving the files correctly.

BTW, I opened one of imap-backup's .mbox files with the Apple Mail program, but didn't see any examples of '- From' or any other incorrect caracters.

LMRW commented 1 month ago

It's bizarre isn't it - sorry I haven't been able to help track down the reason thus far. I'll update if I figure anything out. Thanks again

joeyates / imap-backup

"From:" header is changed to "-From:" #181