The Manhattan Plot is too slow

[pjvandehaar]

Even with a binning threshold of pval=1e-4, the variant bins are the slow part (~500ms).

Improving the code speeds things up, but I think we're mostly constrained by the cost of drawing ~50k circles in the DOM.

Ways to bin variants:

Option 1: When there's a string of neglog10_pvals, replace them with a [start, end] in the json and a line with stroke-linecap:round in the svg. I suspect that this will eliminate 80% of circles.

Option 2: The circles have radius 2.3. We have ~1000 position bins in a plot >=700px wide. Columns overlap their neighbors at browser widths < 3200px. Exploit that.

[dtaliun]

Do you use clipping?

On Apr 16, 2016, at 3:18 PM, Peter VandeHaar notifications@github.com wrote:

Even with a binning threshold of pval=1e-4, the variant bins are the slow part (~500ms).

Improving the code speeds things up, but I think we're mostly constrained by the cost of drawing ~50k circles in the DOM.

So, ways to bin variants:

Option 1: When there's a string of neglog10_pvals, replace them with a [start, end] in the json and a line with stroke-linecap:round in the svg.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/statgen/pheweb/issues/13

[pjvandehaar]

You mean like <clippath>? Not yet.

[dtaliun]

yes, i had an impression that it may slow down drawing speed on some browsers…

On Apr 16, 2016, at 3:39 PM, Peter VandeHaar notifications@github.com wrote:

You mean like ? Not yet.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/statgen/pheweb/issues/13#issuecomment-210880518

[pjvandehaar]

Interesting. Actually, I don't have any need for <clippath>, but I'll have it by default when/if I start using LZ. I think the problem is just having 60k circles in the DOM. Right now http://browser.sph.umich.edu:5000/pheno/362.31 is using 600MB of RAM. :\

[dtaliun]

What about avoiding translate for each group, and instead precomputed coordinates before drawing?

On Apr 16, 2016, at 3:46 PM, Peter VandeHaar notifications@github.com wrote:

Interesting. Actually, I don't have any need for , but I'll have it by default when/if I start using LZ. I think the problem is just having 60k circles in the DOM. Right now http://browser.sph.umich.edu:5000/pheno/994 http://browser.sph.umich.edu:5000/pheno/994 is using 350MB of RAM. :\

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/statgen/pheweb/issues/13#issuecomment-210882648

[dtaliun]

Sorry, I see that you are already doing this

On Apr 16, 2016, at 3:57 PM, Daniel Taliun dtaliun@umich.edu wrote:

What about avoiding translate for each group, and instead precomputed coordinates before drawing?

On Apr 16, 2016, at 3:46 PM, Peter VandeHaar <notifications@github.com mailto:notifications@github.com> wrote:

Interesting. Actually, I don't have any need for , but I'll have it by default when/if I start using LZ. I think the problem is just having 60k circles in the DOM. Right now http://browser.sph.umich.edu:5000/pheno/994 http://browser.sph.umich.edu:5000/pheno/994 is using 350MB of RAM. :\

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/statgen/pheweb/issues/13#issuecomment-210882648

[pjvandehaar]

Well, maybe.

[pjvandehaar]

I used option 1 and now:

• rendering time is <200ms.
• document.querySelectorAll('svg *').length is <10k.
  • cat 362.31.json | sed -n '1,/variant_bins/p' | gzip -c | wc -c is 90KB.
  • cat 362.31.json | grep 'e-05' | wc -l is 5600.
  • cat 362.31.json | grep 'e-' | wc -l is 8400.
  • cat 362.31.json | sed -n '/variant_bins/,$p' | gzip -c | wc -c is 15KB.
• memory usage in Chrome Task Manager (without DevTools) is <200MB.

Look at http://browser.sph.umich.edu:5000/pheno/797.1 from 10^-4 to 10^-5. It's sparse, but apparently contains many variants. Should I "de-dupe" the unbinned variants? Or just the pvals worse than 10^-5? Or bin all but the top 4k pvals?

Maybe add a tooltip for binned variants showing their position, to make the cutoff less jarring?