Closed zz85 closed 8 years ago
zz85
commented
8 years ago Turns out that <webview> can also run in a separate process and can access the file system (with modemodules="on") https://github.com/atom/electron/blob/master/docs/api/web-view-tag.md
Might just try that out too
zz85
commented
8 years ago some updates after some experiments:
No go for IndexedDB, it probably also have a limit
The serialized value is too large (size=166082386 bytes, max=133169152 bytes).
Chunking updates seems to incur some amount of JS processing and blocking on the ui too, I'm not too favourable of it now.
@wizztjh also recommends ServiceWorker cache, which seems like an alternative to explore.
WebView is a go, it's probably easier to control a webview than a separate renderer process. However, WebView IPC still imposes some message limits. Which comes down to the last options
After various testing on speeding up scanning the disk, I'm settling with simply using the node.js async FS methods. However, running the disk intensive methods seems to starve the libuv event loop a little way that even setIntervals seems to be heavily delayed. What's worse the experience of running electron's renderer process while readDirs would seem like the UI has became unresponsive.
My current approach (#5) to that is to run disk scanning in separate (headless) renderer process (either spawn from browser or remote). However, for the target UI process to receive data, the disk scanning processing would need to send messages as an electron IPC method. The scan disk process would send the browser (main.js) an IPC message, and the browser would send the UI process another IPC method. At the expense of more responsive UI overall, it seems the overheads are the price to pay for message passing, and the UI would still seem to freeze marshalling a large json.
One of my favourite hacks for doing message passing on the web is abusing localStorage. LocalStorage is usually pretty performant and allows multiple pages to listen for localStorage change events and that allows me to write a simple localStorage IPC mechanism. Without any benchmarks, this would be faster than electron's IPC in theory (since it would be serialize once instead of twice), and it feels faster than with electron's IPC (yes, feeling is subjective and bad for actually measuring performance).
There's a catch with localStorage, because there're limits associated with how large a string you could write to it.
"Uncaught QuotaExceededError: Failed to set the 'lsipc' property on 'Storage': Setting the value of 'lsipc' exceeded the quota."A simple workaround for now is simply to fallback to electron's IPC calls when localStorage throws me an error (see #7). However even electron IPC has it limits when I start getting
[50582:1023/134245:ERROR:ipc_channel_reader.cc(65)] IPC message is too bigin my logs.What other choices do we have? Right out at the top of my mind is the following
IndexedDB seems like a natural upgrade for localStorage. I haven't use that, but it seems like the next natural path to take. Web Workers could be a lighter form of spawning a headless process, but doing
require('fs')in a Web Worker seems like a no-no for now. Which then the next better thing would be to try service workers.Apart from trying to incase the amount of data that can be transferred, the existing size limitation could be overcome is messages were sent in chunks. One way is perhaps to send diffs for updating in a way a streaming JSONParser could recreate the JSON message entirely.
Lastly, the simplest UI hack would be actually how I think most scanning applications does that. Simply not do anything or show a progress bar until the entire scanning is done (then the disk scanning could be done in the same process).