search

UPennEoR / ml_rfi

Machine learning applied to RFI flagging
BSD 2-Clause "Simplified" License
8 stars 4 forks source link

Latest real data training dataset (v3) has bad RFI flags #3

Closed jkerrigan closed 5 years ago

jkerrigan commented 6 years ago

There's quite a few waterfall visibilities where major RFI like ORBCOM has been missed by the standard XRFI flagger. These bad waterfalls need to be pruned as they ultimately confuse the FCN in training.

jkerrigan commented 6 years ago

Up to waterfall visibility 900 appears to be decently flagged. Everything beyond has huge issues where ORBCOM is completely passed over.

jkerrigan commented 6 years ago

This evaluation dataset actually needs to be overhauled entirely, poor accuracy and recall is reported because XRFI contains flagged band edges, and without proper band edge modeling the FCN won't try to flag these parts. The dataset is also heavily weighted to a single observation (across multiple baselines).

jkerrigan commented 5 years ago

This dataset has been deprecated, with a newer version that has better vetting.