Calculate DIA Object statistics

[troyraen]

We should calculate some statistics on DIA "objects", store them in a BigQuery table, and incorporate them into the pipeline.

Here are some thoughts:

Columns to calculate

• magnitude stats per band: min, max, average
• earliest and latest detection
• average ra and dec
• list of sourceIds contributing to the object

Need to think about when/how to update it.

• If it's reasonable to query BigQuery live for every single alert, consider adding the table update in the _storeBigQuery pipeline component. (waiting for implementation/results from #100).
• Could also do it once every morning. Collecting metadata generates a list of alerts processed overnight (could also query BQ for this info).

Incorporate into pipeline

• Some value-added component should query the table and add the object info to the alert packet. If we do this, might as well (1) update the object stats with the new alert info; (2) publish the value-added message to Pub/Sub; (3) update the BigQuery table
• If we want to use the object info later in the pipeline, it needs to be a component that publishes a stream that actually gets listened to (i.e., not in _storeBigQuery).

Other Brokers:

Most (all?) do this in a similar way. Here is an example from Lasair:

The objects table contains metadata for groups of candidates with the same object identification number, assigned by ZTF to all candidates whose positions agree within 1farcs5. This metadata includes minimum, maximum and average magnitudes, earliest and latest dates of detection, and mean coordinates. We create an object when there are three or more candidate detections, in order to remove moving objects and reduce bogus detections. Therefore all the ZTF object metadata is stored in the objects table only if there are three or more detections within 1farcs5.

[troyraen]

Note that BigTable may be better suited to this. It may provide a more efficient means to both update the data on an ongoing basis and to query it on a per-alert basis for cross-matching. This table could also store cross match and classification data for the object. BigTable is good for very wide tables with groups of columns (e.g., for a single cross match) structured in different ways.