search

joeyates / imap-backup

Backup and Migrate IMAP Email Accounts
MIT License
1.37k stars 75 forks source link

write the metadata file once after refreshing all flags #200

Closed devinlane closed 4 months ago

devinlane commented 5 months ago

The DelayedMetadataSerializer works excellently during message download, but the FlagRefresher uses Serializer without any batching or transaction, causing the metadata file to be written for each message. For my backup of ~50k messages, this produces a rather large amount of disk writes (~20GB for INBOX alone).

I've updated the FlagRefresher to do the refresh work inside a Serializer transaction, which I've implemented to do transactions on both mbox and imap metadata.

In my testing this makes a incremental mirror backup take < 1 min with almost no disk writes.

joeyates commented 4 months ago

Hi @devinlane

Thanks for the PR, it addresses an area that wasn't previously handled - slow flag refreshes.

I decided to follow you lead, but to implement in a different way, by moving the flag refresher call inside the existing transaction block. This way it follows the existing 2 strategy approach.