emirkmo
commented
3 years ago Preliminary working version here: https://github.com/emirkmo/flows/tree/wcs_fix now @sholmbo is working on replacing the astroalign calls with a custom wcs class that can do rotation etc. It's working quite well already using just blind nearest neighbor search. When he has an update, we should merge and increment version as it's a big step.
Outstanding questions:
- How to keep track of the new/re-calcualted wcs? Perhaps in a separate fits file tagged with the image name and pipeline version or just add it to photometry.ecsv or photometry.log?
- Balance between checking and warning for bad WCS vs. trying to correct for it. We need to establish this.
- What metric to use to track wcs improvement? Right now using reference star matches. Can we have a global metric like nreferences expected vs nreferences found that is calculated after the fact?
- Unit test of some sort to avoid false corrections. Best way is to have a trackable metric.
It would be great if flows did a basic WCS check, and even tried to correct for it if the photometry failed.
I think we should update run_photometry.py to try to automatically correct the WCS if the photometry fails. In that case, the new WCS would also need to be added to the image header. We shouldn't do this unless the photometry fails, and we should still require users to submit images with correct WCS. But sometimes it's obviously just a little bit wrong, and it would be possible to deal with this automatically.
I propose we use some combination of sep and astroalign/reproject for this. The problem with daofind is needing the FWHM estimate, and it not performing well on variable background as one gets from stacked & dithered NIR images. sep.extract can deal with it using matched filtering and should be quite fast.
If we use sep to do the source detection, then we can calculate offsets and reproject the image using e.g. astroalign, and saving a copy, or adjusting the WCS. Using the matched filtering algorithm of sep has the additional benefit that we can then check those sources using our forced 2d gaussian fitting and clean bad stars, etc. and get a very good FWHM estimate. At that point, we remove the daofind steps from the pipeline, make it possible to maybe fix WCS when it is a bit off, and keep our bad star rejection in place, without adding a lot of overhead.
Moving to sep for doing the initial source extraction has an issue in that the Cython code underneath can sometimes fail in not a very nice way, (for example running out of hard-coded memory allocation or dividing by zero) which we would have to catch. Anyway, I think it's worth testing out because it can give us a better bad star rejection and make it possible to do a basic WCS check, and even a basic WCS correction, on all the images by default.