Out of Memory issues for large datasets + large gene selections

[JobLeonard]

I have been running into allocation bugs:

What you're looking at is the browser complaining through my console that it doesn't have enough memory to allocate arrays for all the genes.

Now, this happens when I visit multiple 10k+ sized loom files after another, and request a few hundred genes in each. So not a worry at the moment

If we want to scale to 100k or 1 million cells this will start happening with a handful of genes, so it is something to anticipate for later.

There are two main reasons this happens:

• wasteful use of arrays. They're being allocated and thrown away all over the place (even after I've optimised a large amount of these things away). Any redraw of the sparklines generates an entire new array for each gene, for example.
• we store all loaded datasets in memory. We could load/unload datasets into offline storage when not being viewed (issue #54) to alleviate this. To do so we need to move the datasets out of Redux, however.

Fixing this will be a painful process, and is not necesary right now. It should be done at some point, however.

Roadmap:

• [ ] split dataset storage out of redux
  • [ ] redux should only contain view logic, really
  • [ ] for the datasets that have been moved out of the redux store, let it remain mostly immutable (the data doesn't change, after all), but allow for in-place mutations in select places for more optimal memory usage. Notably preparing the data for plotters
  • [x] write more memory-efficient plotters (should also improve performance a bit more again, yay)
  • [ ] for datasets that are not currently being viewed, unload them from memory, and store them in local storage (if an update is necessary). As a side benefit, this would implement off-line storage of loom access, which is a nice feature to have anyway.

[JobLeonard]

Note: partially fixed by not fetching all genes at once