joeyates
commented
1 year ago Hi @garymoon
This looks promising in helping work towards a workaround for this problem.
I'll message you via your commit email.
Closed garymoon closed 10 months ago
joeyates
commented
1 year ago Hi @garymoon
This looks promising in helping work towards a workaround for this problem.
I'll message you via your commit email.
garymoon
commented
1 year ago Thanks for looking into this Joe! I've replied to your email with unredacted output :+1:
joeyates
commented
1 year ago @garymoon I've pushed a pre-release with a fix for this: 9.4.0.pre1
Please give it a try!
Note that this is for the character encoding problem only. The log shows a failed fetch at the end (UID 39910), for which I think there is nothing to be done.
patric-r
commented
11 months ago It still happens in 11.0.1.
E, [2023-08-04T19:21:44.791071 #2081] ERROR -- : Backup for account 'user@domain.com' failed with error incompatible character encodings: UTF-8 and ASCII-8BIT
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/email/mboxrd/message.rb:31:in `+'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/email/mboxrd/message.rb:31:in `to_serialized'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/delayed_metadata_serializer.rb:35:in `append'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:95:in `handle_uid_and_body'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:78:in `block in download_block'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:77:in `each'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:77:in `with_index'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:77:in `download_block'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:32:in `block in run'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:31:in `each'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:31:in `each_slice'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:31:in `with_index'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/downloader.rb:31:in `run'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/account/backup.rb:76:in `block in backup_folder'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/delayed_metadata_serializer.rb:25:in `block (2 levels) in transaction'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/mbox.rb:19:in `block in transaction'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/transaction.rb:17:in `begin'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/mbox.rb:18:in `transaction'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/delayed_metadata_serializer.rb:24:in `block in transaction'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/transaction.rb:17:in `begin'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/serializer/delayed_metadata_serializer.rb:23:in `transaction'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/account/backup.rb:75:in `backup_folder'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/account/backup.rb:38:in `block in run'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/account/backup_folders.rb:39:in `block in each'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/account/backup_folders.rb:39:in `each'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/account/backup_folders.rb:39:in `each'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/account/backup.rb:37:in `run'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/cli/backup.rb:29:in `block in run'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/cli/backup.rb:27:in `each'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/cli/backup.rb:27:in `run'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/cli.rb:72:in `backup'
/home/user/.local/share/gem/ruby/3.2.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
/home/user/.local/share/gem/ruby/3.2.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
/home/user/.local/share/gem/ruby/3.2.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
/home/user/.local/share/gem/ruby/3.2.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/cli.rb:45:in `start'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/bin/imap-backup:12:in `block in <top (required)>'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/lib/imap/backup/logger.rb:41:in `sanitize_stderr'
/home/user/.local/share/gem/ruby/3.2.0/gems/imap-backup-11.0.1/bin/imap-backup:11:in `<top (required)>'
./imap-backup:25:in `load'
./imap-backup:25:in `<main>'
@joeyates I can provide you a test imap account which immediately reproduces the problem. If you're interested, post your mail address I will send you the account details.
joeyates
commented
11 months ago Hi @patric-r
Could you contact me via the email address that shows up in my commits?
Thanks
joeyates
commented
10 months ago I've released version 11.1.0.rc1 with a fix for this problem. @patric-r please give it a try!
patric-r
commented
10 months ago @joeyates
The new 11.1.0.rc1 works like a charm and fixes the issue. I've tested it on several mailboxes containing a large amount of mails and I was now able to backup everything without any errors.
Thank you for your effort!
MatejKovacic
commented
10 months ago When the new version will be pushed out?
joeyates
commented
10 months ago @MatejKovacic I've just released version 11.1.0
MatejKovacic
commented
10 months ago Thanks a lot! Works like a charm.
Hello,
I believe I have a reproducible case for this issue. It's worth noting that this also breaks sync with Thunderbird and isync/mbsync.
Below is the
-voutput (semi-anonymized with[snip]) ofimap-backupattempting to sync a directory with a message in it that appears to break Dovecot. This pattern will repeat with a small pause between attempts. There are three such messages in my mailbox.For anyone else encountering this issue: To temporarily work around this I added instances of
uids.delete([UID])at the top ofruninlib/imap/backup/downloader.rbfor each offending message.@joeyates I'm happy to share verbatim copies of the bad messages privately, directly copied from the server's maildir, but of course there's no guarantee this condition will be reproducible even with the verbatim messages. I'm happy to get you any further information you'd like, and apply any code changes for testing, but I would like to remove the bad messages as soon as I can so I have a working mailbox again haha :sob:
Some unsolicited ideas that would make problems of this type less of an issue:
grep -vto get to this stage, in particular the search responses are messy in a terminal)Thank you muchly for imap-backup! :blue_heart:
imap-backupoutputIncluding the isync/
mbsyncerror here for Googlers since it's vastly easier to work around inimap-backup.