Use Precomputed Visits and Summary Stats.

Quansight / lsst_dashboard

LSST Dashboard https://quansight.github.io/lsst_dashboard/

BSD 3-Clause "New" or "Revised" License

8 stars 3 forks source link

Use Precomputed Visits and Summary Stats. #165

Closed dharhas closed 4 years ago

dharhas commented 4 years ago

closes #154
closes #153
closes #112
re-enables coadd forced/unforced datasets (not fully tested)
uses summary stats for ranges and percentiles instead of calculating them
cleans up cli
moves all precompute into partition.py

Currently works with the following datasets:

/project/dharhas/DM-21335-New-KTK

and

/project/dharhas/DM-21335-New-KTK-1Perc

These were created with:

lsst_data_repartition /project/tmorton/tickets/DM-21335-new/pdr2/deep/xmm /project/dharhas/DM-21335-New2-KTK --queue=normal --nodes=6

and

lsst_data_repartition /project/tmorton/tickets/DM-21335-new/pdr2/deep/xmm /project/dharhas/DM-21335-New2-KTK-1Perc --sample_frac=0.01 --queue=normal --nodes=6

Notes:

this requires https://github.com/dask/distributed/pull/3704
you will need to rerun pip install --user -e ./ since the cli has changed

timothydmorton commented 4 years ago

Testing this out, first thing I've noticed is that if I load a single metric in skyplot view, and then switch to detail view, it still takes ~1m for the detail view plots to appear. CPU is at 100%, and there are very sparse blips of dask activity during this time.

timothydmorton commented 4 years ago

This is what the dask graph looks like when loading the detail view.

There seem to be 2x20 sets of mini 'getitem' calls in series. I can't figure out where this is coming from, except that all the wait seems to be happening right before the last line of _switch_view_mode() when it is called when going to the detail view for the first time. The delay is definitely not coming from constructing the visit-timeseries plot now, which is great, but we need to figure out where this is coming from. @brendancol @dharhas @philippjfr does anyone see this pattern of dask tasks when you run things locally on first switch to detail view after loading one metric?

timothydmorton commented 4 years ago

And just confirmed that the speed of this scales with data size--that is, the 1Perc dataset breezes quickly through that section of the graph, but with the same structure.

dharhas commented 4 years ago

@timothydmorton I saw the same behavior related to the mini 'getitem' calls but didn't get around to looking into what was causing it. I need to zip up the 1perc dataset so @philippjfr & @brendancol can test offline.