joeyates / imap-backup

Backup and Migrate IMAP Email Accounts
MIT License
1.33k stars 74 forks source link

Migrate stops without error #203

Open martinpichlo opened 3 weeks ago

martinpichlo commented 3 weeks ago

Hi,

I've backuped some email accounts via backup command in mirror mode. For testing purpose I tried to restore some bigger accounts (more than 5 GB) via migrate command into a temporal test account. Unfortunately it aborts, although the current folder and a lot of others folders, were not restored. Sometimes with and sometimes without any error message.


First try: Stop's without any error message. I noted, that the uid e.g. 10278 is not in sync with the counter e.g. 10271/13587.

~/imap-backup $ ./imap-backup.sh migrate source@domain.de test@domain.de --reset --verbose ... D, [2024-06-09T07:46:07.867108 #1] DEBUG -- : [Gesendet] uid: 9230 (9225/13587) - 76045 bytes .... ... D, [2024-06-09T07:53:15.433479 #1] DEBUG -- : [Gesendet] uid: 10277 (10270/13587) - 4070463 bytes D, [2024-06-09T07:55:28.537342 #1] DEBUG -- : [Gesendet] uid: 10278 (10271/13587) - 1414207 bytes ~/imap-backup $


Second try: With second --verbose

~/imap-backup $ ./imap-backup.sh migrate source@domain.de test@domain.de --reset --verbose --verbose ... C: C: S: 9992 EXISTS S: 9992 RECENT S: RUBY10671 OK [APPENDUID 1717705200 31382] Append completed (0.098 + 0.217 + 0.096 secs). D, [2024-06-08T21:47:49.482746 #1] DEBUG -- : [Gesendet] uid: 10000 (9993/13587) - 6286394 bytes ~/imap-backup $


Third try: Also I tried to restart the migration. For that I used the same migrate command, without reset option. But in that case, it's ends with duplicate mails. For example my archive contains 661 emails, and 1322 emails afterwards.

~/imap-backup $ ./imap-backup.sh migrate source@domain.de test@domain.de --verbose .... D, [2024-06-08T19:22:03.015870 #1] DEBUG -- : Creating IMAP instance: domain.com, options: {:port=>993, :ssl=>{:min_version=>771}} D, [2024-06-08T19:22:03.288483 #1] DEBUG -- : [Archiv] 661 to migrate D, [2024-06-08T19:22:03.288974 #1] DEBUG -- : [Archiv] uid: 1 (1/661) - 40732 bytes D, [2024-06-08T19:22:03.774696 #1] DEBUG -- : [Archiv] uid: 2 (2/661) - 41652 bytes D, [2024-06-08T19:22:04.050382 #1] DEBUG -- : [Archiv] uid: 3 (3/661) - 37830 bytes


PS: Note: I'm using imap-backup within docker container including a wrapper script. I will do some tests without docker as next step.

Who can help?

martinpichlo commented 3 weeks ago

I also run into trouble on my MacBook, where I've installed imap-backup via home-brew. After the very huge sent folder with over 13.000 emails, were uploaded, it hangs up in the next folder. Now the program just hangs, so that I need to cancel it after long time without any progress.

.... D, [2024-06-09T19:42:45.692934 #80238] DEBUG -- : [INBOX/Avisen] uid: 56 (56/244) - 44285 bytes D, [2024-06-09T19:42:45.977402 #80238] DEBUG -- : [INBOX/Avisen] uid: 57 (57/244) - 36568 bytes D, [2024-06-09T19:42:46.261995 #80238] DEBUG -- : [INBOX/Avisen] uid: 58 (58/244) - 18641 bytes D, [2024-06-09T19:42:46.553522 #80238] DEBUG -- : [INBOX/Avisen] uid: 59 (59/244) - 31520 bytes D, [2024-06-09T19:42:46.846000 #80238] DEBUG -- : [INBOX/Avisen] uid: 60 (60/244) - 74774 bytes

joeyates commented 3 weeks ago

Hi @martinpichlo

Thanks for opening this issue.

I suspect that there is a large email after uid 60 that is taking a long time to upload. Could you check network activity?

Also, please activate network logging with --verbose --verbose to see if there are any clues there.

martinpichlo commented 3 weeks ago

Hi @joeyates,

I will double check the case with uid 60.

Meawhile: Do you may have ideas regarding the aborts in my initial question? I have used --verbose --verbose there.