search

vladimiry / ElectronMail

Unofficial ProtonMail Desktop App
GNU General Public License v3.0
1.51k stars 99 forks source link

Avoid loading local store on startup? #468

Closed CodeCracker-oss closed 2 years ago

CodeCracker-oss commented 2 years ago

Hello,

as I use the local store for being able to view past emails when theres no internet, its not something I use on a day to day basis, but I don't want to disable it.

Can you implement a feature where user can avoid loading local store after entering password (ie local store can be loaded when actually switch to it)

Every time i start electronmail it takes roughly 1-2 minutes to load, which is inconveinient.

vladimiry commented 2 years ago

Sure it's technically possible to do but would take more work than the original/current implementation. I'm not planning to step in into this task in the near future.

Every time i start electronmail it takes roughly 1-2 minutes to load, which is inconveinient.

This looks like a lot. What is the database file size, CPU and storage type (HDD/SSD)?

CodeCracker-oss commented 2 years ago

@vladimiry

What is the database file size, CPU and storage type (HDD/SSD)?

Database File Size: 174.8MB (183,331,591 bytes)

CPU: AMD Ryzen 5 4600H

SSD: Samsung SSD 980 PRO 2TB (SSD is a Gen4, though motherboard only supports up to Gen3)

vladimiry commented 2 years ago

Thanks. I remember I tested it on a very ancient laptop on a 50MB DB file size (low voltage intel i5 4th gen) and loading was taking like 3-4 seconds. I don't have a large DB file at the moment to test/debug things but I think loading just 174.8MB DB in 1-2 minutes is not ok.

vladimiry commented 2 years ago

I think I might want to make these values configurable so I could ask someone with a large DB file to tweak the config file and see how it changes things.

@dhammel, did you happen to remember if it worked faster before https://github.com/vladimiry/ElectronMail/releases/tag/v4.12.2 / Jun 02, 2021? Since v4.12.2 the app started processing the database file by chunks to reduce memory use peaks/max.

CodeCracker-oss commented 2 years ago

@vladimiry

I remember when you had updated it before, right after it started showing the text "Loading loal store" it started taked longer, but that was right when I restore a lot of messages to protonmail from my prior gmail account when switching permentaly over to protonmail, so I believed that to be the cause due to an increase in file size but maybe it's not. You say it took 3-4 seconds for a 50mb database, so triple that to equal almost the size of mine would take ~16 seconds.

vladimiry commented 2 years ago

You say it took 3-4 seconds for a 50mb database, so triple that to equal almost the size of mine would take ~16 seconds.

Yep, and divide the value then to the i5-4th-gen vs ryzen-4600h performance difference index :smile: I didn't run tests but I believe that mail count size also contributes to the speed drop since we get more chunks, etc (currently chunk size is defined by 800-1000 mails portion).

CodeCracker-oss commented 2 years ago

@vladimiry

did you happen to remember if it worked faster before https://github.com/vladimiry/ElectronMail/releases/tag/v4.12.2 / Jun 02, 2021?

I am not sure if that tag is the one I referred to in above comment, or not. If not, then I can't recall if the issue occured with it or not. The only way I could test would be to downgrade, and start from scratch as its mentioned its not backwards compatible.

I believe that mail count size also contributes to the speed drop since we get more chunks

Well, if it helps. According to Thunderbird (All Mail section), I have a total of 6.328 conversations and 1,088 conversations in another e-mail account (both of which are added in electronmail, synced with local store).

vladimiry commented 2 years ago

@nadinethebrain I remember you had 250 MB database file size after switching to v4.12.2+ (so after it got compressed, etc). Does the app start take long (that "loading database file ..." process/gray-progress-line after master password submitting)?

THe only way I could test would be to downgrade, and start from scratch as its mentioned its not backwards compatible.

Sure that would be too much to do. I just was curious maybe you remember.

I have a total of 6.328 conversations and 1,088 conversations in another e-mail account (both of which are added in electronmail, synced with local store).

And mails count (not conversations)? That 50MB database file on weak laptop stored just about 5500 mails.

CodeCracker-oss commented 2 years ago

@vladimiry

Does the app start take long (that "loading database file ..." process/gray-progress-line after master password submitting)?

Yes, after entering password and submitting it'll greying out and show "Loading "local store" ..." for about 1-2 minutes.

And mails count (not conversations)?

Sorry, I didn't think to try in electronmail itself but according to electronmail there is 6,771 in 1 account, and 1,136 in another total of: 7,907 emails stored in it's database.

I do send attachments often in my e-mails, which could account for a bigger than normal filesize.

(Don't know how to check the individual e-mail count in thunderbird gui, dont have much experience with thunderbird.)

vladimiry commented 2 years ago

Yes, after entering password and submitting it'll greying out and show "Loading "local store" ..." for about 1-2 minutes.

I got it but also asked @nadinethebrain to share the experience.

Sorry, I didn't think to try in electronmail itself but according to electronmail there is 6,771 in 1 account, and 1,136 in another total of: 7,907 emails stored in it's database.

So just ~8k in summary which doesn't look like a lot.

CodeCracker-oss commented 2 years ago

@vladimiry

I got it but also asked @nadinethebrain to share the experience.

My bad, read through it little too quick.

So just ~8k in summary which doesn't look like a lot.

Definately not as muh as other's who rely heavily on email (Most my e-mails are from companies. and such), it probably has to do with attachments being added to so many e-mails, though there faily low in size due to e-mail restrictions, like ~10MB or less, usually in the kb. Electronmail doesn't handle attachments, right? If so, then attachments probably aren't the concern for slower loading then.

I will probably end up just disabling local store mode, as mentioned I don't use it that often. Will probably just revert to thunderbird when needing access to e-mails offline. Even if it took only ~20 seconds to open, it's still a little more than I'm willing to wait considering how little I actually use local store mode.

vladimiry commented 2 years ago

The DB file is currently encrypted with sodium's "crypto_secretbox_easy" function (it's hardcoded). I tried the node's "crypto" module using "aes-256-cbc" algorithm and got ~3.5x performance gain. So I think I should:

arch-btw commented 2 years ago

@vladimiry

I noticed something that might help. My startup time was 1 minute and 20 seconds (sometimes more).

So I went to settings and they were in this state:

2

So I changed it to:

1

Now startup is only 5 seconds.

Here's the odd thing though: when I change it back to the original default (secretbox_easy), it's still only 5 seconds. So the speed improvement remains now.

Did it maybe unintentionally optimize my database or something?

vladimiry commented 2 years ago

Did it maybe unintentionally optimize my database or something?

@arch-btw what you changed only affects the settings.bin key derivation & encryption processes. That file is the one which actually needs these settings as the key derivation is a thing that is primarily responsible for speed vs encryption strength balancing. It's unclear why "sodium.crypto_secretbox_easy" is/was slow on your system but it appears that the issue can be closed with a "glitch" label. As said before I'm not going to debug the case at this time. Besides I need a huge DB file for proper testing which I currently doesn't have at my disposal.

CodeCracker-oss commented 2 years ago

I just tried changing Encryption preset to node.crypto aes-256-cbc and I can load local store within 5 seconds, much faster. Changing back to default seems to stay within the same 5 second load time, odd indeed. However for now at least I'm keeping the non-default one enabled.

arch-btw commented 2 years ago

Thank you vladimiry and thank you @dhammel for testing.

After a reboot of my computer the startup time from ElectronMail went back to 1 minute 20 seconds again.

Another thing I found is that changing the API entry point from .ch to .com also reduces the startup time back to 5 seconds.

So it appears that changing unrelated settings has this effect on startup times. However, it does not persist after a computer reboot.

I understand you don't have time for this, it's no problem. I just thought I'd give you an update.

Thank you for the hard work ! :+1:

vladimiry commented 2 years ago

Another thing I found is that changing the API entry point from .ch to .com also reduces the startup time back to 5 seconds.

This action implies re-saving the settings.bin file. So I guess changing the account order (via drag-n-drop) or doing any other account-related change will also make a similar effect.

After a reboot of my computer the startup time from ElectronMail went back to 1 minute 20 seconds again.

To clarify, if you restart the app multiple times it's still ok (fast), but after the computer restart it becomes slow again (sounds odd indeed)?

The insight into why this oddness could occur has not come to me yet. I've never faced such issue at my side.

arch-btw commented 2 years ago

This action implies re-saving the settings.bin file.

Yes I think this is why.

To clarify, if you restart the app multiple times it's still ok (fast), but after the computer restart it becomes slow again (sounds odd indeed)?

Yes correct.

I just tested it again:

  1. Startup time was 5 seconds because I changed the api point.
  2. Completely exit and re-open electronmail several times ----> still 5 seconds.
  3. Reboot computer
  4. Start electronmail -----> now 1 minute 20 seconds.
  5. Change api point from .com to .ch again (or vise versa) and the boot up time for electronmail is now 5 seconds again.
vladimiry commented 2 years ago

Can you trigger settings.bin re-save by the action other than changing the API entry point (like re-ordering the accounts, changing account alias, any other account-related update - when you click the "update" button on the account edit form)?

I think I will be making a test app build with the change described in https://github.com/vladimiry/ElectronMail/issues/468#issuecomment-983401806 so you could switch to "aes-256-cbc" for DB file encryption.

vladimiry commented 2 years ago

Can someone test the build uploaded here (build job)? This build makes the app apply the encryption settings you choose in the app settings to all the encrypted files (not only to settings.bin file encryption). So you can select aes-256-cbc and see if it speedups things. Besides that, the build comes with updated @protonmail clients.

CodeCracker-oss commented 2 years ago

@vladimiry I can verify, that I was having the same issue as @arch-btw and this test build, after switching to and applying aes-256-cbc seems to so far load local store in exactly 5 seconds, much faster than before. It persist along reboots of pc as well.

Do you know why exactly this occurred? I'm curious

Thank you for all your work :)

arch-btw commented 2 years ago

Thank you @vladimiry it works now 👍 Great job.

CodeCracker-oss commented 2 years ago

@vladimiry I should note that during further testing, when changing encryption preset back to secretbox_easy on the test build, I get an error about 5 seconds through:

Invocation timeout of calling "reEncryptSettings" method on "electron-mail:ipcMain-api" channel with 2500ms timeout.

vladimiry commented 2 years ago

Do you know why exactly this occurred? I'm curious

I don't. The "sodium.crypto_secretbox_easy" encryption for some reason is abnormally slow in some cases (on some systems). Besides based on reports posted here there is some sort of persisted "cache" which gets reset between reboots.

I get an error about 5 seconds through

Good finding. The timeout value was originally designed for settings.bin re-encryption only, but now I added DB files re-encryption to the same "reEncryptSettings" action. So I will be increasing the timeout value.

Besides that I'm going to change the default encryption function to "aes-256-cbc" so at least new average users won't face the issue like this.

vladimiry commented 2 years ago

So I will be increasing the timeout value.

'm going to change the default encryption function to "aes-256-cbc" so at least new average users won't face the issue like this.

Booth points have been handled, so closing the issue.

vladimiry commented 2 years ago

By the way, with #387 enabled it won't be possible to load the "local store" only when you actually switch to the respective view mode since some stuff from the store gets requested by #387 feature in the background mode.