Sparkline heatmap: normalise genes, allow for clamping range

[JobLeonard]

Requested by Amit

[JobLeonard]

Don't laugh, it's been a while since I did this kind of stuff :P http://www.d.umn.edu/~deoka001/Normalization.html

[slinnarsson]

Another request:

Instead of two heatmap color ranges, replace one with a binary option (e.g. white/red) to more clearly shows zeros and non-zeros.

/Sten

-- Sten Linnarsson, PhD Professor of Molecular Systems Biology Karolinska Institutet Unit of Molecular Neurobiology Department of Medical Biochemistry and Biophysics Scheeles väg 1, 171 77 Stockholm, Sweden +46 8 52 48 75 77 (office) +46 70 399 32 06 (mobile)

On 31 May 2017, at 10:32, Job van der Zwan notifications@github.com wrote:

Don't laugh, it's been a while since I did this kind of stuff :P http://www.d.umn.edu/~deoka001/Normalization.html

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[JobLeonard]

I think we will get that for "free" once we introduce clamping: just set the upper bound of the clamping range to 0%. Or do I misunderstand the request?

[JobLeonard]

Side-track: flame map and how to do the heat maps.

So while I was implementing the flame map code I noticed an oddity in the heat map code: if the biggest outlier was non-zero, it would display that, and otherwise it would display the mean value. This has the effect of greatly increasing noise in the heat map plots, and suggesting a lot more signal then there really is.

What I think I intended to do was take the average of the two (remember that the algorithm we use for max outlier selects alternating outliers), compromising between noise profile and average.

I made a comparison overview below. I suggest opening them in new tabs, so you can quickly switch between them for comparison.

I used the Oligos All set, which is 200k cells, so we have around 200 cells per vertical column of pixels (which itself is 40 pixels, which kind of shows you the problems of accidentally hiding data that we are dealing with here).

First is a set sorted by clusters but semi-random otherwise, and then one sorted by total (so one should expect a more smooth gradient).

The plots are:

• Bar (remember that our bar plots show alternating outliers in black, average value in grey)
• Flame
• Heatmap (average of outlier + mean)
• Heatmap (mean value only)

Sorted by clusters

Bars

Flame map

Heat map (average of greatest outlier + mean value)

Heat map (mean value)

Sorted by total

Bars

Flame map

Heat map (average of greatest outlier + mean value)

Heat map (mean value)

I think it's safe to say that the mean-value-only heat map smooths things out too much, and flame maps are better in every way compared to that.

However, when data is sparse there are only a few non-zero values, flame maps seem to have a similar problem to averaging away the non-zero values (see the upper six of the shown genes). The heat map that averages out the outlier/mean value is still useful in that situation, although they give the false impression of a bigger signal than there really is.

So I'm thinking of mixing the two: underneath the flame map, I will put a thin strip of heatmap. It's pretty clear which is which, but it will help spot the outliers a lot more. Here is a mock-up:

Again, feedback is requested!

(I'm also wondering if the bar plots are really giving the proper impression of the data (funny how the flame map helps me reach that conclusion here))

[JobLeonard]

Problem: due to rounding, we can have two different bin sizes. For large datasets (like the Oligos one) this isn't really an issue - 100 or 101 cells doesn't make a big difference. For small datasets it leads to situations like this:

Some bins have two cells, some bins have three. This is a quite a significant difference in vertical height, as well as signal.

I think the best solution is to always plot the vertical graph as if it was the biggest bin size (in this case, as if there are three cells in it - effectively padding bins with two cells with a zero value).

[JobLeonard]

Results of fixing the bin size looks a bit funky, but in a way is also more clear about what the issue is.

What the data looks like for a small dataset, showing an attribute that we also sort by (so a smooth gradient should be the result), at various zoom levels:

1 or 2 cells per column:

2 or 3:

4 or 5:

A slightly larger dataset (6000+ cells, so 4 or 5 cells per column), sorted by k-means as primary key, log-cv as secondary, and log-mean as tertiary:

Forebrain set (44k cells, so somewhere around 25 cells per bin)

For large enough datasets the problem simply vanishes, because we reach the point where there are multiple cells per pixel:

[JobLeonard]

Gioelle suggested that flame maps might also be useful for smaller datasets when printing, since those plots are often only a few centimetres across. This photo from a page in Erik Smedler's PhD thesis provides a quick example (although the time-vs-activity plots themselves are not directly comparable, obviously):

So as a quick test, I opened the cortex.loom set (3005 cells), resized the browser window, and took screenshots comparing flame maps to bar plots and heatmaps. Since the latter two are using the LTTB outlier-selection algorithm in our browser I also made a quick temporary version of Loom where they display the mean values instead, to see how their smoother output compares to the flame graph:

[JobLeonard]

@gioelelm suggested that we log-transform the data before determining the heatmap scale (specifically, log2(1 + value - minValue)). Below are some quick tests, it's should obvious which is with the projection and which is without it:

Anyway, Gio was pretty happy with it. The only thing that needs to be decided if whether we want log-scale to be on or off by default (Gio wants it on by default).

• [x] add log projection as option to color selection code
• [x] add toggle to interface
  • [x] scatterplot
  • [x] sparklines
  • [x] heatmap
• [x] change heatmap legend in scatterplot to show when log is enabled
  • [x] change label text
  • [x] change colour gradient