TraP appears to "skip" Sources in bad Images

[ycendes]

Strangely enough, in my latest database (which crashes eventually, but that's another issue filed earlier) we appear to have a new issue in that some of the previously extracted sources are being "skipped" by TraP. See this number of sources plot to see what I mean: http://banana.transientskp.org/master/vlo_cendesrfitest/dataset/10/

What we currently know:

• The "gaps" in the number of sources appear to happen when we head from a good image to a bad one- here is one of the first examples to give you an idea- http://banana.transientskp.org/master/vlo_cendesrfitest/image/760/ (Ultimately, TraP finally crashes when it encounters one of these bad images.)
• These gaps are also really visible if you look at the running catalogs for different sources. As in, here is one source (that appears to be Cas A/Cyg A, which is why it's likely still existing in the high-RFI images)- http://banana.transientskp.org/master/vlo_cendesrfitest/runningcatalog/83/ To compare, this is the running catalog for a source that "skips" the bad images- http://banana.transientskp.org/master/vlo_cendesrfitest/runningcatalog/117/ Note, there definitely ARE images in the gaps, but it's not showing up with all the extracted sources!

All I have for now, I can't think of any other simple immediate things to check.

[AntoniaR]

If you are force fitting to a specific Gaussian shape (i.e. force-fit to restoring beam), here's what I think is happening (I've seen this in my MWA datasets):

PySE often fails to fit sources when force-fitting. This can be mitigated against by adapting the sourcefinder settings, but it's virtually impossible to completely prevent in the force-fit strategy. Basically, it fails to find a good fit to the source and rather than outputting rubbish it just ignores the source.

It's easy to check this as TraP outputs all the failed fit locations when it processes an image (not sure if this is in the saved log, but it's certainly on screen). The way I would probably test this is by setting a ridiculously high detection threshold and then put one of the sources that often fails into the monitoring list - then you only have one source to think about and TraP should be much faster.

[gijzelaerr]

The RMS of the images where the forced fits fails is extremely high. These images would be filtered out when we implement a proper RMS check for AARTFAAC. Something AARTFAAC users can do manually in their images_to_process.py file.

Not sure if this a bug or not.

[AntoniaR]

If my explanation is correct (we need to check the TraP logs to confirm) then TraP is behaving as expected and this is not a bug.

[gijzelaerr]

ok closing this issue, please reopen if you think otherwise.