JobLeonard
commented
7 years ago See, I already made a mistake in the previous scheme: we're not going to filter per attribute, but per unique value in an attribute. So Class is an attribute, oligodendrocytes a unique value within this array. We filter on the latter.
Also, if one would filter out all oligo's in the Class attribute, then obviously all associated subclasses in values in Subclass would also be filtered out. It is probably useful to keep track of this, so that we, for example, can hide invisible items from the legend as well.
{
dataSets: {
[dataset]: {
filteredRowIndices: [0, 3, 4, 5, 14, 15, …],
rowAttrs: {
'(original order)': [0, 1, 2, 3, …],
[rowAttribute]: {
data: ['array', 'with', 'attribute', 'data', …],
mostFrequent: [
{
val: 'most common value',
count: /* nr of occurrences */,
filtered: false,
visible: true,
},
{
val: 'second most common value',
count: /* nr of occurrences */,
filtered: false,
visible: true,
},
…
],
colorIndices: /* hasmap of the twenty most
frequent values as keys,
with value being an index
for the color LUTs */
}
}
}
}
}
As always, this is self-documentation and Rubber Duck Debugging/Architecture Planning. This is just a plan, things will change as I implement it, but since it touches almost every part of the client-side code-base I want to write it out first.
So this actually grew out of of having to fix that colouring bug in the metadata views. Currently we do a lot of recalculations of information that is not changing.
For example, the row and column metadata will not change for a given dataset. Yet every time we render a sparkline or scatterplot we calculate the 20 most frequent values on the fly, then determine the colours. This is wasteful, but ok: in my tests so far, even mobile browsers barely hiccup on this. Similarly, the way I have implemented sorting the data in sparklines is on the fly, every time the sparkline view is rendered.
We can do better, and probably simplify and clean up the code in the process.
What I want to do is calculate all of this information once, then let the views just display static data whenever possible. This should be done at the point when a dataset has been successfully fetched (or in the case of sorting/filtering: whenever the relevant setting changes).
For sorting/filtering, we'll store the original row and column orders as simple arrays (
[0, 1, 2, 3, ..]). For values that are filtered out, we copy the array to afilteredIndiceswithout the indexes of the values filtered out (I've already made this, btw, it's just not used anywhere yet).The next step is to store the keys we sort by and in what order:
We can implement stable sort by moving a selected sort key to the front of the array, and then sort
filteredIndicesusing a custom comparison function that goes through the array.In all of our views where we display row- or columndata, we then pass these filtered indices along with the data, then make a copy of the data using filtered indices:
Then we use
dataSubsetwhere we would use the originaldatabefore.As for colour selection, we'll just calculate
nMostfrequentfor each row/column attribute when we receive the dataset, store that, and create a lookup hashtable for the colours.So the relevant part of the redux state tree will change as follows:
(changes are same for column attributes)
Note that I added an
(original order)attribute. This is a simple trick to make sorting by original order trivial: just put that "attribute" to the front of the list of sort keys; means we can simplify our algorithm for that.This will require some deep rewrites in the plotting functions and the views, but the result should be a much more elegant code base (maybe smaller, even!), and easier to follow data structures in the redux side. It will also make sharing settings between the views easier, or sometimes even happen for free: since whether or not a row attribute is filtered will become part of the row attribute, that setting will be automatically shared between the views.