maritz / culling-stats-webapp

MIT License
3 stars 1 forks source link

Chrome crashes on large data sets #2

Open maritz opened 8 years ago

maritz commented 8 years ago

Reason for the crash for large amounts of data: The parsing is all done in a separate thread with a webworker. Currently it just gathers ALL the results and puts them into a giant object that is then sent to the main thread via postMessage. This kills the chrome. With a big enough data set I expect it will probably kill anything you throw it at.

The planned fix: Each time a file is parsed, send it down to the main thread, keep a compiled and normalized store of the important data there and update it.

Added benefits from that method:

maritz commented 8 years ago

I've changed it so that games get passed down to the main thread as soon as they are parsed and the huge entries array is not needed anymore.

This decreases the parse time quite a bit, especially during the 99% phase.

In theory it should also reduce the chance for a crash. Since I haven't had the log amount to produce the crash in the first place, I cannot confirm this though.