Implement metrics to measure scientific performance of classifiers participating in ELAsTiCC. By the end of the sprint we should make a collection of scripts / notebooks / DESC TOM portal integrations to query for the current metrics values and graphs.
Resources and skills needed
Basic Python skills are required, SQL knowledge is good, but not necessary.
[ ] Per class completeness and purity as a function of redshift, and pertinent truth table properties per class (e.g. looking at variable star completeness vs true period)
[ ] Distinguishing SN-like objects from others (this is different from SN-classification, as we only care if the object is flagged as an SN of any type, rather than e.g. AGN)
[ ] Broad anomaly detection - can object be identified as anomaly-like any of KN/SLSN/TDE/CART/ILOT/PISN (i.e. you can label a true PISN as an ILOT and it still counts because it was still flagged interesting)
[ ] Accurate anomaly classification - accuracy for same classes as broad anomaly detection. Note that LIGO Skymaps are available for some of the KN but not all LSST KN will be seen in LIGO.
[ ] Early classification - time from first alert for classifier score to exceed a threshold of 0.75, evaluated in two windows (1 week after first alert, and 1 week before true peak). Again here we will use SN-like Do we have peak time in the database?
[ ] Number of observations needed to identify an object as a periodic variable, and number of observations needed to flag which kind of periodic variable accurately.
ELAsTiCC: performance metrics for classifiers
Development and implementation of classification metrics for ELAsTiCC
Contacts: Konstantin Malanchev kostya@illinois.edu Day/Time: Oct 20-21 (Thu-Fri) Main communication channel: #plasticc-public channel of the LSST slack GitHub repo: https://github.com/LSSTDESC/elasticc_metrics Zoom room (if applicable): https://stanford.zoom.us/j/903652385 (pinned at the channel)
Goals and deliverable
Implement metrics to measure scientific performance of classifiers participating in ELAsTiCC. By the end of the sprint we should make a collection of scripts / notebooks / DESC TOM portal integrations to query for the current metrics values and graphs.
Resources and skills needed
Basic Python skills are required, SQL knowledge is good, but not necessary.
Detailed description
The metrics we consider to implement:
[x] Confusion matrices, implemented as a script https://github.com/LSSTDESC/elasticc_metrics/blob/6b3e4d49da9359ead8e7d9872e9d6aefb66b1d4e/sql_query_conf_matrices_objects.py
[ ] Metrics from @aimalz paper, see https://github.com/aimalz/proclam and Section 3 of https://ui.adsabs.harvard.edu/abs/2019AJ....158..171M/abstract
[ ] Per class completeness and purity as a function of redshift, and pertinent truth table properties per class (e.g. looking at variable star completeness vs true period)
[ ] Distinguishing SN-like objects from others (this is different from SN-classification, as we only care if the object is flagged as an SN of any type, rather than e.g. AGN)
[ ] Broad anomaly detection - can object be identified as anomaly-like any of KN/SLSN/TDE/CART/ILOT/PISN (i.e. you can label a true PISN as an ILOT and it still counts because it was still flagged interesting)
[ ] Accurate anomaly classification - accuracy for same classes as broad anomaly detection. Note that LIGO Skymaps are available for some of the KN but not all LSST KN will be seen in LIGO.
[ ] Early classification - time from first alert for classifier score to exceed a threshold of 0.75, evaluated in two windows (1 week after first alert, and 1 week before true peak). Again here we will use SN-like Do we have peak time in the database?
[ ] Number of observations needed to identify an object as a periodic variable, and number of observations needed to flag which kind of periodic variable accurately.