Support for Search message content

[ystxn]

Is there a reason why this feature is hidden in ElectronMail?

The local store is incredibly slow to index and seems rather inefficient - taking days on 30k convos. It also slows down startup to a crawl. I'd much rather use a native proton feature if that works more efficiently.

[vladimiry]

Some reasons listed in #466. Closing as duplicate.

[vladimiry]

The local store is incredibly slow to index

Currently, the index is not stored but gets built on the fly, when the app starts. Indexing was fast enough on ~5k messages test executed on an old/slow ULV CPU. But at some point it might be transformed to a stored index. I suspect it's not the indexing performance issue you experience, but the initial data loading/decrypting.

[ystxn]

Mine seems to have stalled after a few hours and i've kept it going for days. The loading icon is still blinking and the database view mode shows only 11k messages in All Mail when it should be ~30k. Also takes almost a minute for Electron to start up now.

Conversely, Proton's index took only 10 minutes and the search works really well. Pity that it can't be used. Looks like I'll have to jump to browser if I need to use search.

[vladimiry]

If ~30k loading didn't get finished, then it's not indexing, but the data fetching case. I won't be able to look into it since at the moment I don't have a similar dataset at my side.

Conversely, Proton's index took only 10 minutes

To please the API rate limiting set by proton at the server-side, the app comes with a hard rate limit put on the API calls at the app side (275 calls per minute, by default). It's unknown to me what would be a more relaxed rate limiting options (can be revealed by experimenting), but Proton being the backend owner knows the exact rate limiting options, so it's obvious that data loading is going to work faster for them.

Another possible reason is that they/proton might be using a new bulk-like API to fetch the messages vs individual calls used in the app since Jan 2019 / https://github.com/vladimiry/ElectronMail/releases/tag/v2.2.0. I didn't look yet into what kind of API calls they are using to fetch the messages for building a local search index.

[vladimiry]

Fixing potential issue of blocked initial local store data load (bootstrap fetch) in the next release.

The speed I got on ~6 mails account is ~80 messages per minute, so 30k messages processing would take ~6 hours. Proton sets Priority: 'u=7' request header for API calls related to the encrypted search. I guess this header value allows bypassing the rate limiter. I might test it in action one day.

[vladimiry]

Next release gets the following "local store" related update:

• Improve the local store feature:
  • Make the bootstrap fetch process resumable. The saving progress up to the saving portion size might still be lost and so redownloaded on resuming (500 messages by default), but not the entire progress.
  • Show basic progress indication as an account handle title (on mouse hover).