search

joeyates / imap-backup

Backup and Migrate IMAP Email Accounts
MIT License
1.33k stars 74 forks source link

incompatible character encodings: UTF-8 and ASCII-8BIT #173

Closed michaldejmek closed 7 months ago

michaldejmek commented 9 months ago

Hello,

imap-backup 12.0.0 - incompatible character encodings: UTF-8 and ASCII-8BIT:

/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/encodings.rb:137:in `join'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/encodings.rb:137:in `value_decode'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/encodings.rb:109:in `decode_encode'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/fields/unstructured_field.rb:84:in `do_decode'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/fields/unstructured_field.rb:62:in `decoded'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/fields/unstructured_field.rb:66:in `default'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/field.rb:239:in `method_missing'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/message.rb:1215:in `default'
/var/lib/gems/3.1.0/gems/mail-2.7.1/lib/mail/message.rb:666:in `from'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/email/mboxrd/message.rb:75:in `first_from'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/email/mboxrd/message.rb:66:in `best_from'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/email/mboxrd/message.rb:59:in `from'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/email/mboxrd/message.rb:31:in `to_serialized'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:46:in `to_serialized'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:30:in `block in append'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:52:in `block (2 levels) in rollback_on_error'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/mbox.rb:19:in `block in transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/transaction.rb:17:in `begin'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/mbox.rb:18:in `transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:51:in `block in rollback_on_error'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/imap.rb:30:in `block in transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/transaction.rb:17:in `begin'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/imap.rb:29:in `transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:50:in `rollback_on_error'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:29:in `append'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer.rb:93:in `append'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:95:in `handle_uid_and_body'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:78:in `block in download_block'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:77:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:77:in `with_index'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:77:in `download_block'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:32:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `each_slice'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `with_index'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/folder_backup.rb:31:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer.rb:37:in `transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/folder_backup.rb:30:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup.rb:35:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup_folders.rb:39:in `block in each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup_folders.rb:39:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup_folders.rb:39:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup.rb:34:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli/backup.rb:29:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli/backup.rb:27:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli/backup.rb:27:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli.rb:72:in `backup'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli.rb:45:in `start'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/bin/imap-backup:12:in `block in <top (required)>'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/logger.rb:41:in `sanitize_stderr'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/bin/imap-backup:11:in `<top (required)>'
/usr/local/bin/imap-backup:25:in `load'
/usr/local/bin/imap-backup:25:in `<main>'"
 (RuntimeError)
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:34:in `rescue in block in append'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:30:in `block in append'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:52:in `block (2 levels) in rollback_on_error'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/mbox.rb:19:in `block in transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/transaction.rb:17:in `begin'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/mbox.rb:18:in `transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:51:in `block in rollback_on_error'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/imap.rb:30:in `block in transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/transaction.rb:17:in `begin'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/imap.rb:29:in `transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:50:in `rollback_on_error'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer/appender.rb:29:in `append'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer.rb:93:in `append'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:95:in `handle_uid_and_body'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:78:in `block in download_block'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:77:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:77:in `with_index'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:77:in `download_block'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:32:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `each_slice'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `with_index'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/downloader.rb:31:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/folder_backup.rb:31:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/serializer.rb:37:in `transaction'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/folder_backup.rb:30:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup.rb:35:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup_folders.rb:39:in `block in each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup_folders.rb:39:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup_folders.rb:39:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/account/backup.rb:34:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli/backup.rb:29:in `block in run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli/backup.rb:27:in `each'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli/backup.rb:27:in `run'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli.rb:72:in `backup'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
/var/lib/gems/3.1.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/cli.rb:45:in `start'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/bin/imap-backup:12:in `block in <top (required)>'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/lib/imap/backup/logger.rb:41:in `sanitize_stderr'
/var/lib/gems/3.1.0/gems/imap-backup-12.0.0/bin/imap-backup:11:in `<top (required)>'
/usr/local/bin/imap-backup:25:in `load'
/usr/local/bin/imap-backup:25:in `<main>'
joeyates commented 9 months ago

Hi @michaldejmek

From the backtrace it looks like the message that is causing the problem is malformed, and the Mail gem is unable to parse it.

In order to understand what's going I'd need to have the original message. You can send it to me at imap-backup@MY_GITHUB_USER_NAME.info

joeyates commented 9 months ago

@michaldejmek

I received the email, thank you. Unfortunately, I am unable to reproduce the error using the text of that message.

Please run imap-backup again with the --verbose flag and send me the output.

michaldejmek commented 9 months ago

@joeyates This mailbox is over 100GB :( It won't be easy to do - it takes 2 days to export. Can I back up only the selected directory .?

joeyates commented 9 months ago

Backing up just the selected directory is fine

joeyates commented 9 months ago

Hi @michaldejmek

Were you able to re-run this backup with --verbose?

joeyates commented 8 months ago

@michaldejmek Any news?

michaldejmek commented 6 months ago

@hi @joeyates

Sorry for the late reply. "Incompatible character encoding: UTF-8 and ASCII-8BIT:" is the same error I sent you in the email. This is a newsletter - the solution would be to not stop the process and just note this error and move on. If I used the "-verbose" command, the process ran to completion.

Personally, I think the email messed up the encoding when it was created and sent.

Personally, I would like the program to automatically create a log for each mailbox and the log would include the name of the mailbox, how many emails are in it, how many were transmitted and how many were bad. Possibly log the errors in a log.