d3v-null
commented
2 years ago I agree: we should add a GPSSTART key
could add a fits table of flag occupancy for each channel.
Closed cjordan closed 2 years ago
d3v-null
commented
2 years ago I agree: we should add a GPSSTART key
could add a fits table of flag occupancy for each channel.
MWA legacy observation 1090008640 has Cotter flag files (a.k.a. mwaf files) with the key "GPSTIME = 1090008640". Birli currently produces mwaf files that also have this GPSTIME. However, this time does not correspond to any of the gpubox HDU GPS times. Greg believes that this is a rare example of gpubox HDUs being written at times at an offset to the scheduled start time. Obviously if we can't tell what the first timestep of the flags was derived from, there's an ambiguity on what the flags should be applied to.
I don't believe there has been much formalisation on the mwaf file format, so this is an opportunity to tighten things.
I think there are two paths to take with GPSTIME - leave it as the obsid, or make it the first GPS timestamp of the contained flags (we cannot assume that all the flags in an mwaf file are derived from all the available data for that observation). I am in favour of the second option, as having an OBSID key is clearer. On the other hand, GPSSTART (or similar) would also be clear for when the flags start. Maybe we can have both and leave GPSTIME has a historic quirk.
I've looked at the RTS source and it does not use GPSTIME (reads it, but does not use it). I think the RTS assumes that there are enough flags for all data timesteps, and it assumes the first flag timestep corresponds to the first data timestep. So I don't think we need to consider the RTS here; there should be a lot more rigour around what it's doing.
Finally, RTS can take advantage of keys structured "REFLG_{:02}" (e.g. REFLG_00) to indicate flags that have a very high occupancy (e.g. in this case, channel 0 is flagged say >80%). Obviously the threshold is a variable, but the default for RTS utilities is 80%, and I think that's sensible. It would be nice to have these automatically included in Birli mwaf files.