search

joeyates / imap-backup

Backup and Migrate IMAP Email Accounts
MIT License
1.33k stars 74 forks source link

What means that .mbox file is longer than indicated by .imap file? #169

Closed MatejKovacic closed 9 months ago

MatejKovacic commented 9 months ago

I say imap-backup local check and for some folders I get this error:

INBOX/...: .mbox file '/home/matej/.imap-backup/ ... .mbox' is longer than indicated by .imap file '/home/matej/.imap-backup/ ... .imap'

Also, I run imap-backup today and got this error (and that was the reason why I run local check command):

I, [2023-09-14T15:05:48.600816 #259920]  INFO -- : [INBOX/***] 2 new messages
E, [2023-09-14T15:05:51.866658 #259920] ERROR -- : Backup for account 'matej***' failed with error closed stream
/usr/lib/ruby/3.1.0/openssl/buffering.rb:354:in `syswrite'
/usr/lib/ruby/3.1.0/openssl/buffering.rb:354:in `do_write'
/usr/lib/ruby/3.1.0/openssl/buffering.rb:453:in `print'
/home/matej/gems/gems/net-imap-0.3.4/lib/net/imap.rb:2239:in `put_string'
/home/matej/gems/gems/net-imap-0.3.4/lib/net/imap.rb:2211:in `block in send_command'
/usr/lib/ruby/3.1.0/monitor.rb:202:in `synchronize'
/usr/lib/ruby/3.1.0/monitor.rb:202:in `mon_synchronize'
/home/matej/gems/gems/net-imap-0.3.4/lib/net/imap.rb:2206:in `send_command'
/home/matej/gems/gems/net-imap-0.3.4/lib/net/imap.rb:1118:in `block in examine'
/usr/lib/ruby/3.1.0/monitor.rb:202:in `synchronize'
/usr/lib/ruby/3.1.0/monitor.rb:202:in `mon_synchronize'
/home/matej/gems/gems/net-imap-0.3.4/lib/net/imap.rb:1116:in `examine'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/client/default.rb:62:in `examine'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/folder.rb:152:in `examine'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/folder.rb:35:in `block in exist?'
/home/matej/gems/gems/imap-backup-11.1.0/lib/retry_on_error.rb:4:in `retry_on_error'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/folder.rb:34:in `exist?'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/folder_backup.rb:41:in `folder_ok?'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/folder_backup.rb:23:in `run'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/backup.rb:35:in `block in run'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/backup_folders.rb:39:in `block in each'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/backup_folders.rb:39:in `each'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/backup_folders.rb:39:in `each'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/account/backup.rb:34:in `run'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/cli/backup.rb:29:in `block in run'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/cli/backup.rb:27:in `each'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/cli/backup.rb:27:in `run'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/cli.rb:72:in `backup'
/home/matej/gems/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
/home/matej/gems/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
/home/matej/gems/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
/home/matej/gems/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/cli.rb:45:in `start'
/home/matej/gems/gems/imap-backup-11.1.0/bin/imap-backup:12:in `block in <top (required)>'
/home/matej/gems/gems/imap-backup-11.1.0/lib/imap/backup/logger.rb:41:in `sanitize_stderr'
/home/matej/gems/gems/imap-backup-11.1.0/bin/imap-backup:11:in `<top (required)>'
/home/matej/gems/bin/imap-backup:25:in `load'
/home/matej/gems/bin/imap-backup:25:in `<main>'
joeyates commented 9 months ago

Hi @MatejKovacic

That does sound like that mailbox is corrupt.

I suggest renaming the .mbox and .imap files and running another backup.

The error you are seeing during backup is unrelated as it a problem with accessing a remote folder.

MatejKovacic commented 9 months ago

So the error was: INBOX/Arhivi/isju: .mbox file '/home/matej/.imap-backup/.../isju.mbox' is longer than indicated by .imap file '/home/matej/.imap-backup/isju.imap'

I went to this folder and renamed those two files:

I run imap-backup local check and there was no error. Then I run imap-backup again and got:

I, [2023-09-19T10:04:27.120641 #420190]  INFO -- : Running backup of account: ***
I, [2023-09-19T10:04:40.717797 #420190]  INFO -- : [INBOX/Arhivi/isju] 100 new messages

Right after that I run imap-backup local check again, and the same error is back: INBOX/Arhivi/isju: .mbox file '/home/matej/.imap-backup/.../isju.mbox' is longer than indicated by .imap file '/home/matej/.imap-backup/isju.imap'.

joeyates commented 9 months ago

@MatejKovacic I've found the source of the problem. It was a bug in the handling of character encodings, so it affects all people who have emails with non-ASCII characters.

Please upgrade to version 12.0.0 and rerun your backups.

Thanks for opening this issue as it allowed me to find an important bug.

MatejKovacic commented 9 months ago

I upgraded to the v. 12, but I get the same error when I run imap-backup local check. This is the size of the files:

ls isju.* -lha
-rw-rw-r-- 1 matej matej 5,8K sep 20 08:27 isju.imap
-rw-rw-r-- 1 matej matej 1,2M sep 19 10:04 isju.mbox
joeyates commented 9 months ago

Hi @MatejKovacic

I didn't make it explicit that you need to rename or move the old backups before re-running imap-backup. Did you do so?

If you did, then I'll need to look into the problem again.

Please let me know

MatejKovacic commented 9 months ago

OK, I did that, and now it says everything is OK.

However, when I deleted the files and ran the imap-backup, new mail were downloaded, but I got some errors:

WARNING: Ignoring unparsable header "osnutku. Kon=C4=8Dni produkt naj bi bila priporo=C4=8Return-Path: <lugos-list-bounces+***=***@lugos.si>": invalid header name syntax: "osnutku. Kon=C4=8Dni produkt naj bi bila priporo=C4=8Return-Path"
WARNING: Ignoring unparsable header "s.si/cgi-bin/mailman/listinfo/lugos-list>, \r\n\t<mailto:lugos-list-request@lugos.si?subject=unsubscribe>": invalid header name syntax: "s.si/cgi-bin/mailman/listinfo/lugos-list>, \r\n\t<mailto"
WARNING: Ignoring unparsable header "2A85665E120\r\n\tfor <lugos-list@fo.lugos.si>; Fri, 28 Jan 2022 13:44:21 +0100 (CET)": invalid header name syntax: "2A85665E120\r\n\tfor <lugos-list@fo.lugos.si>; Fri, 28 Jan 2022 13"
WARNING: Ignoring unparsable header "for <Return-Path: <lugos-list-bounces+***=***@lugos.si>": invalid header name syntax: "for <Return-Path"
I, [2023-09-20T13:37:33.600222 #488071]  INFO -- : [INBOX/***] 740 new messages
WARNING: Ignoring unparsable header ":0:0:0:182 (LHLO ***.***.si)\r\n (*:*:*:*:*:*:*:*) by ***.***.si with LMTP; Wed, 19 Jan 2022\r\n 09:03:12 +0100 (CET)": invalid header name syntax: ""
I, [2023-09-20T13:41:07.290378 #488071]  INFO -- : [INBOX/***] 7447 new messages

Any idea what should that mean? Are e-mail correctly backuped anyway?

joeyates commented 9 months ago

Hello again. Those are warnings due to malformed email headers. Backups are stored even if emails are malformed.

julpec commented 9 months ago

I have the same problem with WARNING: Ignoring unparsable header except that I'm trying to migrate. Since the headers are invalid, they are not exported correctly and the emails comes without subjects, recipients, etc... However, the headers are in the body of the mail. Quite annoying and unusable Thanks

joeyates commented 9 months ago

Hi @julpec

Please open an issue indicating what happens during your migration.

Please note, though, that in a migration, imap-backup copies messages to the destination as they are received. Parsing is only done to extract metadata used during storage. If headers are reported as unparseable during backup, it is not surprising that the destination server may not be able to parse them either — the problem lies with the messages themselves.

julpec commented 9 months ago

Hi, Ok, I will open another issue. By the way, there were no errors during the backup phase…

glocknerc commented 9 months ago

I have the same problem as originally reported, .mbox' is longer than indicated by .imap file, but I'm already running version 13. I've deleted accounts and recreated them, deleted the backup directory and ran a backup from scratch, to no avail.

To expand on this a bit:

  1. I'm running it in mirror mode
  2. I start off with an empty storage directory, so no backup yet
  3. I run the first backup, it finishes without errors
  4. I run imap-backup local check, it returns no errors
  5. I immediately runimap-backup again, the output tells me it's deleting e-mails only present locally (even though no e-mail on the server was deleted or otherwise modified since the last backup)
  6. I run imap-backup local check, it returns the error

In case it matters, the account I'm backing up is hosted on iCloud.

Update: Through trial and error I found that it matters which mode I use for saving metadata, so I raised #178