Array buffer allocation failed

nadinethebrain commented 3 years ago

First of all thanks for the project. I really enjoy using the app. That said, I've been getting this error for a couple of days now. Reinstalled the app multiple times and also tried different versions.

OS: Windows 10 Pro 20H2

[error] src\web\browser-window\app\store\reducers\notification.js addNotificationItem { name: 'RangeError', message: 'Array buffer allocation failed', stack: 'RangeError: Array buffer allocation failed\n' + ' at new ArrayBuffer ()\n' + ' at Encoder.resizeBuffer (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:62:27)\n' + ' at Encoder.ensureBufferSizeToWrite (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:58:18)\n' + ' at Encoder.encodeString (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:178:18)\n' + ' at Encoder.doEncode (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:49:18)\n' + ' at Encoder.encodeMap (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:293:22)\n' + ' at Encoder.encodeObject (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:204:18)\n' + ' at Encoder.doEncode (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:52:18)\n' + ' at Encoder.encodeMap (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:293:22)\n' + ' at Encoder.encodeObject (C:\Users\...\AppData\Local\Programs\ElectronMail\resources\app.asar\node_modules\@msgpack\msgpack\dist\Encoder.js:204:18)' }

I guess it has something to do with memory size or a memory leak? Any clue?

vladimiry commented 3 years ago

What is the database.bin file size (file located in the app settings folder)?
How many approximated to thousands email messages you got in your account?
Do you have the local store option enabled?
How much free RAM your computer has before starting the app?
Do you start the app via shortcut created by installer (there is embedded --js-flags="--max-old-space-size=6144" argument)?
Make sure you start the app with --js-flags="--max-old-space-size=6144" argument (also try increasing the 6144 value lets say to 8192).

nadinethebrain commented 3 years ago

Hey there, thanks for the fast reply. Please find my answers below embedded in the quote:

* What is the `database.bin` file size (file located in the [app settings folder](https://github.com/vladimiry/ElectronMail/blob/master/images/open-settings-folder.jpg))? **About 1.1 GB**

* How many approximated to thousands email messages you got in your account? **About 14.500**

* Do you have the [local store](https://github.com/vladimiry/ElectronMail/wiki/FAQ) option enabled? **Yes!**

* How much free RAM your computer has before starting the app? **24 GB**

* Do you start the app via shortcut created by installer (there is embedded `--js-flags="--max-old-space-size=6144"` argument)? **Yes!**

* Make sure you start the app with `--js-flags="--max-old-space-size=6144"` argument (also try increasing the 6144 value lets say to 8192). **Just tried it. Didn't work.  :( Also tried an even higher value. There is a "App not responding" message showing for a second in the header while it tries to load the local database.**

vladimiry commented 3 years ago

If increasing the number in --js-flags="--max-old-space-size=6144 argument didn't help then there will be no easy solution since there will be a need to redesign the local storage. Few options, more possible (both time consuming):

Splitting database.bin to pieces and processing them one by one.
Turning the storage into the SQLite format instead of currently used JSON <=> messagepack serialization. Then working with it similarly to database.bin, ie in in-memory mode so it's still stored as an encrypted blob.

nadinethebrain commented 3 years ago

I was afraid that, if the first solution with increasing the numbers didn't work, there wasn't any quick and easy way to fix this. The options you listed sound like some reasonable way to solve this problem. That said, I've got nothing to do with coding whatsoever. Unfortunately, despite what my username may suggest, that's a closed book to me haha.

If increasing the number in --js-flags="--max-old-space-size=6144 argument didn't help then there will be no easy solution since there will be a need to redesign the local storage. Few options, more possible (both time consuming):
* Splitting `database.bin` to pieces and processing them one by one.

* Turning the storage into the SQLite format instead of currently used `JSON <=> messagepack` serialization. Then working with it similarly to `database.bin`, ie in `in-memory` mode so it's still stored as an encrypted blob.

vladimiry commented 3 years ago

What I did above is listed some options, it doesn't mean I suggested you pick and implement one, but more like was thinking in writing mode. More thought supposed to be put into the issue before jumping into the actual coding. Yes, the listed things are quite technically challenging. I'm not going to work on the store redesigning in the near future, so unfortunately I have to admit that the app, or at least its local store feature, is not suitable for your setup at this time.

vladimiry commented 3 years ago

@nadinethebrain I'm going to make a simple change for now and then will ask you to test the update (asynchronous decoding).

nadinethebrain commented 3 years ago

What I did above is listed some options, it doesn't mean I suggested you pick and implement one, but more like was thinking in writing mode. More thought supposed to be put into the issue before jumping into the actual coding. Yes, the listed things are quite technically challenging. I'm not going to work on the store redesigning in the near future, so unfortunately I have to admit that the app, or at least its local store feature, is not suitable for your setup at this time.

Haha okay. Sorry. :)

@nadinethebrain I'm going to make a simple change for now and then will ask you to test the update (asynchronous decoding).

Of course I will help with everything in my power even if it's just testing updates. Just kindly hit me up once you came around working on this and I'll be on the spot! Thank you very much.

vladimiry commented 3 years ago

@nadinethebrain can you test the build uploaded here?

How to test:

Open the app settings folder using the currently installed app.
Close the app.
In the opened before folder rename database.bin => database.bin.backup and database-session.bin => database-session.bin.backup. This is required for the app to recreate the databases in an updated structure.
Install new app version by the above-provided link. Then start the app.

Since the "local store"-related files are missing (have been renamed/backuped), the app will start the initial/bootstrap messages fetch. You got a lot of emails, so this bootstrap fetch might take hours (you did it once, so should be aware of it, the average speed I think is like 100 messages per minute). The bootstrap messages fetch process can't be speed up since the app has to respect the rate-limiting @ProtonMail has set on API calls.

After the bootstrap messages fetch got completed, try to restart the app several times to see that the app is still able to read the local store. Also try to make some message read/unread or move it to another folder (we need to cause any blinking in green update), so the local store updating kicks-in, then restart the app (on app start the database-session.bin file gets merged into database.bin, so we test it).

About 1.1 GB

Btw, with this update the database.bin file size will be reduced, likely at least 2x times.

nadinethebrain commented 3 years ago

@vladimiry thank you. will do this over the weekend and post the results afterwards!

Yes I remember that it took some time back then but that's perfectly fine.

vladimiry commented 3 years ago

@nadinethebrain I'm going to upload the updated build soon. It should reduce memory use peaks during local store reading/writing. So if you have not yet started testing the previous build, better hold off for a while.

vladimiry commented 3 years ago

The updated build has been uploaded here. I recommend to backup all the files listed in the FAQ (especially those with .bin extension). Other than that the testing steps listed in previous message are correct.

nadinethebrain commented 3 years ago

@vladimiry I testet the first build you provided and am happy to let you know that it also works as intended. As expected it took some hours for the mails to be fetched again.

That said, the database size was reduced by 3/4 to around 250 MB. What kind of sorcery is this? 👍

What should I do now? Go ahead, backup the data and update again?

vladimiry commented 3 years ago

Thanks for the update. Yes just do data files backup and use the new build for a while, no need to re-fetch all the emails again. Just getting some mails update to trigger the local-sync (like receiving new mails, making read/unread, moving between folders, removing, starring, etc), and then restarting the app several times should be enough to test updated logic. If the updated build works well for you then I will be preparing a new release.

size was reduced by 3/4 to around 250 MB. What kind of sorcery is this?

Simply data compression. The compression ratio could be better, but then the process might take significantly more time/memory, so for now the light compression is used (gzip format).

nadinethebrain commented 3 years ago

Okay great. Will happily do this and keep you posted. Thanks a lot!

nadinethebrain commented 3 years ago

@vladimiry Seems to be working as intended. No issues over here yet.

Btw may I kindly ask if there's a way to support you i.e. make a small donation to your work to show some gratitude for the quick fix? I'm really happy to be able to continue using the app.

vladimiry commented 3 years ago

make a small donation

You can subscribe to unresolved #258 and get a notification when the issue gets resolved ("Subscribe" button at the screen right side).

nadinethebrain commented 3 years ago

Okay. Done. 👍

make a small donation

You can subscribe to unresolved #258 and get a notification when the issue gets resolved ("Subscribe" button at the screen right side).

vladimiry commented 2 years ago

@nadinethebrain if you still use the app, can you test some stuff using this build? Asking since I remember you have got a huge database.bin file.

Things to do:

Install that build.
Backup the database.bin and database-session.bin files (just in case, no data corruption expected to happen).
Change the dbCompression value in the config.json in the following way (file located in the app settings folder, you should see similar content in the file if you already switched to v5.0.0 but with "type": "gzip" value ...):
```
"dbCompression": {
"type": "zstd",
"level": 4,
"mailsPortionSize": {
  "min": 550,
  "max": 650
}
}
```

This is a just about switching the compression from gzip to zstd algorithm. The database.bin file size should be reduced to like 25% (likely not instantly, but after some db data change gets triggered and app restart occurs, so the file gets re-saved). If you got a powerful CPU, the "level": 4 value can be increased, to maybe 8, which enables better compression.

There is nothing special to do, just use this build for a while and report if the app crashes or so. The build runs well for me, but the db file is small (few dozens MB).

vladimiry / ElectronMail

Array buffer allocation failed #406