What you're looking at is the browser complaining through my console that it doesn't have enough memory to allocate arrays for all the genes.
Now, this happens when I visit multiple 10k+ sized loom files after another, and request a few hundred genes in each. So not a worry at the moment
If we want to scale to 100k or 1 million cells this will start happening with a handful of genes, so it is something to anticipate for later.
There are two main reasons this happens:
wasteful use of arrays. They're being allocated and thrown away all over the place (even after I've optimised a large amount of these things away). Any redraw of the sparklines generates an entire new array for each gene, for example.
we store all loaded datasets in memory. We could load/unload datasets into offline storage when not being viewed (issue #54) to alleviate this. To do so we need to move the datasets out of Redux, however.
Fixing this will be a painful process, and is not necesary right now. It should be done at some point, however.
Roadmap:
[ ] split dataset storage out of redux
[ ] redux should only contain view logic, really
[ ] for the datasets that have been moved out of the redux store, let it remain mostly immutable (the data doesn't change, after all), but allow for in-place mutations in select places for more optimal memory usage. Notably preparing the data for plotters
[x] write more memory-efficient plotters (should also improve performance a bit more again, yay)
[ ] for datasets that are not currently being viewed, unload them from memory, and store them in local storage (if an update is necessary). As a side benefit, this would implement off-line storage of loom access, which is a nice feature to have anyway.
I have been running into allocation bugs:![screenshot_20170418_104532](https://cloud.githubusercontent.com/assets/259840/25236267/be6d3a9e-25e7-11e7-8493-d54571c05593.png)
What you're looking at is the browser complaining through my console that it doesn't have enough memory to allocate arrays for all the genes.
Now, this happens when I visit multiple 10k+ sized loom files after another, and request a few hundred genes in each. So not a worry at the moment
If we want to scale to 100k or 1 million cells this will start happening with a handful of genes, so it is something to anticipate for later.
There are two main reasons this happens:
Fixing this will be a painful process, and is not necesary right now. It should be done at some point, however.
Roadmap: