search

transientskp / tkp

A transients-discovery pipeline for astronomical image-based surveys
http://docs.transientskp.org/
BSD 2-Clause "Simplified" License
19 stars 15 forks source link

implement multi resolution lightcurves #462

Open gijzelaerr opened 9 years ago

gijzelaerr commented 9 years ago

At the moment we just create keep up adding new extracted sources to the database without a retention policy. This will cause slowdowns and eventually disk fill up.

I think we should implement (optional) multi resolution ligtcurves where a lightcurve is 'compressed' or averaged into a lower resolution after a configurable amount of time. Maybe multiple scales of resolution.

For example RRDtool and graphite apply similar techniques to their metrics:

http://oss.oetiker.ch/rrdtool/ http://graphite.wikidot.com/

timstaley commented 9 years ago

So this is a cool idea, but I think it's important to realise that it's basically an entirely new sub-project, and will likely require splitting TKP into sub-packages, then forking / rewriting the database sub-package.

Here's my reasoning: Ideally, we split out PySE, so you can use that, and probably something like a tkp.utils package that includes DataAccessors, quality-checks and the like. But for the database code you will effectively be starting from scratch schema-wise - you'll definitely be able to re-use some ideas (dataset / runningcatalog / image etc) but all of the extractedsource and variability metrics will have to be rethought because they won't apply to a multi-resolution lightcurve. Likewise you'll need a re-implemented visualization tool.

Accordingly, I'd estimate this as a work-package of maybe 3 to 9 developer-months, or maybe just ¯_(ツ)_/¯