TESS/Kepler plugin magnitude conversion potentially skewed by bad data

[orionlee]

From: https://www.aavso.org/small-caveat-tesskepler-magnitude-conversion

This note is about a small caveat in magnitudes shown for TESS / Kepler data. Empirically, the possible difference is tiny (< 0.0001 magnitude) in SAP_FLUX, so it's likely to be a non issue for most, if not all, cases. I'm posting this note in case the caveat is an issue for some people.

Background: I predominantly use Python / Lightkurve for analyzing TESS data, and notice some small discrepancy in the magnitude I got from my Lightkurve workflow and VStar results. In the handful of samples I looked into, the largest difference is 0.000099 magnitude.

Both VStar and my Lightkurve workflow derive magnitude by converting normalized flux to magnitude, assuming the median flux is the reference TESS magnitude. The discrepancy turns out to be due to the flux included to calculate median:

VStar: get the median of all flux values in the FITS file. My Lightkurve workflow: instead of using all flux values, it ignores the fluxes of possibly unreliable cadences based on the quality flags in the FITS file. For Lightkurve users, ignoring possibly unreliable cadence is the default behavior when reading TESS FITS file (that one can override).

As a result, the derived magnitude is slightly different. The max is 0.000099 mag in the handful of samples (all based on SAP_FLUX).

298734307, sector 57: 0.000022 441794509, sector 57: 0.000003 261136679, sector 1: 0.000015 198537349, sector 23: 0.000055 323968843, sector 39: 0.000019 77209028, sector 4: 0.000099

PDCSAP_FLUX in practice would have no difference most of the time, because most, if not all, likely unreliable cadences have no values in PDCSAP_FLUX to begin with.

For reference: the cadences Lightkurve ignored by default based on quality flag are those with: Attitude Tweak, Safe Mode, Spacecraft is in Coarse Point, Spacecraft is in Earth Point, Reaction Wheel Desaturation Event, Manual Exclude due to an anomaly.

[orionlee]

The issue is probably applicable to Lightkurve plugin (and maybe any other source plugins that derive magnitude by calculating median flux in the data).

[dbenn]

Re: your forum comment "On how to address the issue, I wonder if it might be better to provide an option post load to recalculate median (and thus magnitude)", this could be implemented as an observation transformer plugin, like the existing Magnitude Baseline Shifter and VeLa Observation transformer plugins.

As you also say there, others who have used TESS with VStar may have thoughts about this. I've certainly used TESS with VStar, but others more so. Thoughts: @mpyat2, @clkotnik, @BradWalter?

[mpyat2]

Well, even if it's a minor issue (the difference < 0.0001 has no practical impact), it would be useful to have an observation transformer plug-in, as @dbenn has mentioned.

[orionlee]

If the solution is a plugin akin to Magnitude Baseline Shifter, it could address all sources that derive magnitude by calculating median flux, and handle the potential bad cadences users want to ignore post load.

A minor nuisance in such approach is the plugin needs to know the baseline magnitude to recalculate. For TESS / Kepler case, users could look up the value in, say, https://exofop.ipac.caltech.edu/ , and supply to the plugin. It is probably okay given the recalculation is likely to be needed only in rare cases.

While VStar does know the baseline magnitude for TESS / Kepler case during initial conversion (source) , it is discarded afterwards and cannot be used in recalculation.

[dbenn]

One possibility is that the TESS/Kepler plugins could store the baseline mag as a field in ValidObservation which would be available in the observation table and could be used by an observation transformer plugin. Or it could be supplied by the user as you say.

[orionlee]

This is related, but probably should be tracked in a separate issue.

With PR #316, quality column is included in observations. If an user does want to filter out data by quality flag, the user typically would need to express the condition with bitwise operator. E.g., for TESS, to filter out data points potentially affected by momentum dumps, one would need something like the following:

BITWISE_AND(quality, 32) = 0

It seems that:

• bitwise operator is not supported.
• for QLP, currently quality column is not supported in the VeLa filter, returning an error of unknown binding. (quality is recognized for Kepler/TESS)

[dbenn]

I've added an issue for the bitwise operators @orionlee. If the plugin is rebuilt, a VeLa expression like this:

quality = 512

works in a filter.

[orionlee]

For filter expression such as quality = 512, it works for TESS files, but not for QLP files on my local build from the latest master. For QLP files, I got unknown binding error.

I've tested it on a few different QLP files. For the above screenshot, it is TIC 382517745 sector 39.

To ensure I've got the latest codes, I've done ant clean and verified that the display name for QLP shows v0.3, i.e., including the PR.