search

HERA-Team / hera_qm

HERA Data Quality Metrics
MIT License
2 stars 2 forks source link

Add time_atol to flag_apply to fix XRFI overflagging #409

Closed jsdillon closed 2 years ago

jsdillon commented 2 years ago

@ele-rath and I discovered that one major source of over-flagging in XRFI, as discussed in section 6 of http://reionization.org/manual_uploads/HERA097_H1C_IDR3_2_Memo.pdf, was a bug in XRFI where UVFlag objects created from UVCal objects could not be properly applied to UVData objects when the times were slightly mismatched (an issues we've encountered before).

In our tests, we see rounding errors cause certain integrations identified as bad during the median filter round of XRFI to not get flagged prior to the mean filter round because times were mismatched between the data files and calfits files. In the case of our tests on 2459122, the mismatch was at the ~40 microsecond level. This then led to very bright RFI skewing the statistics of the mean filter and creating edge-effects based on the size of the kernel.

This PR uses the new time_atol parameter in https://github.com/RadioAstronomySoftwareGroup/pyuvdata/pull/1133 to allow times that are close but not perfectly matching to get identified as the same.

Here's a flagging pattern for a subset of that day before this fix:

image

And here's the same pattern after this fix:

image

This PR builds off #408 and requires approval of https://github.com/RadioAstronomySoftwareGroup/pyuvdata/pull/1133 in order for tests to pass.

jsdillon commented 2 years ago

It appears to be the case that instead of implementing a new parameter in pyuvdata, we're instead using the rtol and atol built into the _time_array parameter. In that case, this PR is no longer necessary, since running on the master branch of hera_qm (and the branch of pyuvdata in https://github.com/RadioAstronomySoftwareGroup/pyuvdata/pull/1133) produces the same result:

image