Data quality flags - Githubissues

davecoulter / YSE_PZ

GNU General Public License v3.0

6 stars 15 forks source link

Open astrophpeter opened 2 years ago

astrophpeter commented 2 years ago

YSE_PZ now supports data quality flags. We need to tag data with quality flags, but we want this to be a well defined list of data quality tags.

We need a procedure to add new flags. We need to document these flags

How do we deal with data already in the database? How do we deal with data going forwards?

Let's start the conversation here.

astrophpeter commented 2 years ago

The current flags are:

Quicklook Bad Calibration Crowding Saturation Bad Subtraction Masking Bad

@astrofoley Do we want to keep all of these? For the ones we want to keep could I please have a one liner so I can add them to the documentation.

astrophpeter commented 2 years ago

Proposed new flags by @davecoulter are below, I'll put them as separate comments so people can react and thumbs up or down them.

astrophpeter commented 2 years ago

forced photometry

astrophpeter commented 2 years ago

aperture photometry

astrophpeter commented 2 years ago

psf photometry

astrophpeter commented 2 years ago

unsubtracted

astrophpeter commented 2 years ago

subtracted

astrophpeter commented 2 years ago

I've started the docs page (see here for the draft https://yse-pz--194.org.readthedocs.build/en/194/data_quality.html) in PR #194 . @astrofoley @djones1040 do you have one line descriptions for the current flags? If so reply here and I'll add them to the docs.

cdickinson31256 commented 2 years ago

-maybe better comments about quality, telescope precision/problems, etc throughout the night

astrophpeter commented 2 years ago

Discussion: Tags cannot overlap. Publication quality flag? DOI flag if data has been published?

kywdavis commented 2 years ago

Spectroscopy:

Observation-related Flags:

Conditions (Weather/seeing/etc, maybe we can generalize as "good" or "bad" but we should be consistent)
Bad Cosmic rays ?

Reduction-related Flags: As Peter said, needs to be higher resolution than just "Quicklook"

Same-day cals or archival (most important imo)
Pipeline used (might be mostly redundant if we know who uploaded it)
A billion other reduction steps (host subtracted, telluric corrected, cosmic rays cleaned, optimal vs regular extraction, and probably more). We should think about how to implement this, making n arbitrary flags fo everything is probably bad.