joeyates / imap-backup

Backup and Migrate IMAP Email Accounts
MIT License
1.37k stars 75 forks source link

Backup of a large folder does not work (sent mails, Office 365) #198

Closed Nexulo closed 7 months ago

Nexulo commented 7 months ago

Unfortunately, backups of very large Office 365 "folders" do not work. The folder "Sent items" seems to be over 17GB in size and the backup process aborts every time with the following error logs. Other folders could be backed up without any problems.

The problem is all previously saved mails in this folder are removed because of the crash and the .mbox file contains 0 bytes again. When I start the process again, I do not have to re-authenticate.

So is there any way to solve this problem? Is there a way to cancel the backup process without deleting all previously backed up mails from this folder? This would at least allow me to perform the backup in several steps. I have already tried "ctrl + c", which cancels the script, but unfortunately this also deletes all previously backed up mails from this folder.

Is it even possible to open such a large mbox backup file later to see the mails and attachments?


I, [2024-02-19T10:41:22.295003 #5687]  INFO -- : Running backup of account: mail@example.com
I, [2024-02-19T10:41:24.854208 #5687]  INFO -- : [Gesendete Elemente] 13172 new messages
/usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2809:in `write': closed stream (IOError)
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2809:in `print'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2809:in `put_string'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2781:in `block in send_command'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/3.3.0/monitor.rb:201:in `synchronize'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/3.3.0/monitor.rb:201:in `mon_synchronize'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2776:in `send_command'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:1360:in `login'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/client/default.rb:41:in `login'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/client/default.rb:49:in `reconnect'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/client/automatic_login_wrapper.rb:28:in `method_missing'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:34:in `rescue in run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:21:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/folder_backup.rb:33:in `block in run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/delayed_metadata_serializer.rb:36:in `block (2 levels) in transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/mbox.rb:27:in `block in transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/transaction.rb:26:in `begin'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/mbox.rb:26:in `transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/delayed_metadata_serializer.rb:35:in `block in transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/transaction.rb:26:in `begin'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/delayed_metadata_serializer.rb:34:in `transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/folder_backup.rb:32:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup.rb:35:in `block in run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup_folders.rb:42:in `block in each'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup_folders.rb:42:in `each'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup_folders.rb:42:in `each'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup.rb:34:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/cli/single/backup.rb:38:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/cli/single.rb:183:in `backup'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/invocation.rb:116:in `invoke'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor.rb:338:in `block in subcommand'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/base.rb:584:in `start'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/cli.rb:69:in `start'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/bin/imap-backup:16:in `block in <top (required)>'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/logger.rb:58:in `sanitize_stderr'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/bin/imap-backup:15:in `<top (required)>'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/bin/imap-backup:25:in `load'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/bin/imap-backup:25:in `<main>'
/usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2694:in `get_tagged_response': Session invalidated - AccessTokenExpired (Net::IMAP::ByeResponseError)
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2794:in `block in send_command'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/3.3.0/monitor.rb:201:in `synchronize'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/3.3.0/monitor.rb:201:in `mon_synchronize'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2776:in `send_command'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2858:in `block in fetch_internal'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/3.3.0/monitor.rb:201:in `synchronize'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/3.3.0/monitor.rb:201:in `mon_synchronize'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2853:in `fetch_internal'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/net-imap-0.4.9/lib/net/imap.rb:2080:in `uid_fetch'
    from /usr/local/Cellar/ruby/3.3.0/lib/ruby/3.3.0/forwardable.rb:240:in `uid_fetch'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/client/automatic_login_wrapper.rb:28:in `method_missing'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/folder.rb:94:in `block in fetch_multi'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/retry_on_error.rb:18:in `retry_on_error'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/folder.rb:93:in `fetch_multi'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:61:in `download_block'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:25:in `block in run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:24:in `each'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:24:in `each_slice'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:24:in `with_index'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/downloader.rb:24:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/folder_backup.rb:33:in `block in run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/delayed_metadata_serializer.rb:36:in `block (2 levels) in transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/mbox.rb:27:in `block in transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/transaction.rb:26:in `begin'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/mbox.rb:26:in `transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/delayed_metadata_serializer.rb:35:in `block in transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/transaction.rb:26:in `begin'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/serializer/delayed_metadata_serializer.rb:34:in `transaction'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/folder_backup.rb:32:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup.rb:35:in `block in run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup_folders.rb:42:in `block in each'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup_folders.rb:42:in `each'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup_folders.rb:42:in `each'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/account/backup.rb:34:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/cli/single/backup.rb:38:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/cli/single.rb:183:in `backup'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/invocation.rb:116:in `invoke'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor.rb:338:in `block in subcommand'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/thor-1.3.0/lib/thor/base.rb:584:in `start'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/cli.rb:69:in `start'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/bin/imap-backup:16:in `block in <top (required)>'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/lib/imap/backup/logger.rb:58:in `sanitize_stderr'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/gems/imap-backup-14.6.0/bin/imap-backup:15:in `<top (required)>'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/bin/imap-backup:25:in `load'
    from /usr/local/Cellar/imap-backup/14.6.0/libexec/bin/imap-backup:25:in `<main>'
Nexulo commented 7 months ago

I was able to solve the problem provisionally by setting it to fetch 10 messages at once. However, I ran imap-backup setup to define an account, because I didn't know how to define the fetch-mail size for a single backup. Would that have worked with --fetch-mail 10?

Originally I backed up using the following command:

imap-backup single backup \
  --email mail@example.com \
  --password "" \
  --connection-options '{"port": 1993, "ssl": false}' \
  --server 127.0.0.1 \
  --path backup
joeyates commented 7 months ago

The option to set the number of messages to fetch at a time is --multi-fetch-size= of -n.

You can get this info from the help output:

$ imap-backup help single backup
Usage:                                                                                                                                                
  imap-backup single backup [OPTIONS] 

Options:
<snip>
  -n, [--multi-fetch-size=N]                                                 # the number of emails to download at a time
                                                                             # Default: 1
joeyates commented 7 months ago

In the backtrace you supplied, the backup seems to be failing during login due to an expired access token.

If your backup is failing repeatedly, try turning on debugging output (with --verbose --verbose).

joeyates commented 7 months ago

In order to keep the emails you've downloaded at any point, even after failure, you need to change the "download strategy" from delay to direct.

Note that this makes downloading large mailboxes progressively slower.

Nexulo commented 7 months ago

Many thanks for the help and explanation. The problem has been solved, as I was able to fix it with multi-fetch-size.