Source finding algorithm in tweakreg_catalog is inadequate

[stscijgbot-jp]

Issue JP-2259 was created on JIRA by Howard Bushouse:

It's becoming increasingly clear from reports from the NIRISS, NIRCam, and MIRI teams that there are no good parameters that can be used for source finding in tweakreg in its current form (using DAOStarFinder) that reliably finds only stars without finding lots of garbage. And with the tweakreg algorithm, garbage in leads to garbage out. More often than not, these unreliable catalogs misalign rather than align.

Misty Cracraft, in the #jwst_calpipe_testing slack channel, has reported that there's nothing that works with MIRI imaging data with very clear stellar sources that are (by eye) very easy to detect. The open issue JP-1570 makes the same argument for NIRISS via work Kevin Volk has tried, and James Davies [X] has reproduced. And for NIRCam, I think we are getting a bit lucky with the only tweakreg regression test that we have in our regtest suite, in that it finds lots of garbage, but doesn't result in severe misalignment. I've chatted with Bryan Hilbert (NIRCam) here too, and they just turn it off for simulated mirage data.

DAOStarFinder, which is used in tweakreg_catalog, can find the peaks of JWST stellar PSFs, but it also finds all the chunky bits around it. One can tune this a bit, but it is very sensitive to parameters like kernel_fwhm, threshold, etc. Basically, the Stetson DAOFind algorithm designed for ground-based single-mirror telescopes with CCDs does not work will for JWST. Not a huge surprise.

The fact that it finds multiple sources slightly offset from the actual peak of the star blows up tweakreg. This is the worse possible situation, and results in "reliable" matches to to the offset "sources" and results in very assured misalignment on the part of tweakreg.

Meanwhile, the source_catalog step works great at the end of calwebb_image3. It finds sources reliably with little to no contamination from artifacts. We could just use photutils.segmentation to find sources in tweakreg_catalog. It would be a drop-in replacement, much more reliable, and most importantly, would not depend strongly on initial parameters instrument-to-instrument. A quick check of trying it using the photutils documentation example showed it worked well for the NIRISS example from JP-1570.

The (maybe) downside is that it would also detect non-stellar sources (galaxies), which might not have as precise centroids. But there will be lots of these sources for JWST data, so perhaps the increased number (and lack of proper motion) will mean that the decreased centroid accuracy of individual sources will be swamped by the huge increase in number of sources for extragalactic fields.

Regardless, the calibration pipeline should at a minimum, not make the alignment worse. So we should use an algorithm that at least can reliably detect real sources without detecting junk. Later optimizations to filter those real sources (use only peaky sources with low ellipticity) might offer improvements, but are not necessary now to improve over the status quo where we can not reliably detect sources at all.

Note that large existing surveys using HST already do this. 3D-HST (Candels deep imaging fields + grism) did tweakreg alignment using all sources, not just stars, and did so via SExtractor, i.e. segmentation, and produced mosaics with excellent alignment across multiple instruments. Not many stars in those deep fields. So there's already precedent for this. I suspect other surveys have too.

A possibility for the future, as mentioned by Larry Bradley, would be to use webbpsf and photuils.detection.StarFinder together. I.e. PSF detection. Nobody has tested this yet for JWST, but probably has promise. It would require generating a PSF kernel for each instrument/filter combo (reffile or on-the-fly?) and then using it for detection in photutils. It may be a more optimal solution, but one would have to show that using photutils.segmentation is somehow not performing well in order to justify the added work implementing this. but something to think about and maybe get some INS teams to fiddle with.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

With Ned Molter now assigned to this ticket I've also updated its status in the table on our JWST CalWG Pipeline Enhancement Prioritization Page (in the table that's now current, ie the one labelled "Priority Planning for Jan 2024 onward", I'm also tagging Nadia Dencheva for awareness of this update as well)

[stscijgbot-jp]

Comment by Paul Goudfrooij on JIRA:

HI again Ned Molter - I agree that regression testing is more about ensuring consistency. My comment was in response to your question to Kevin Volk asking for examples that give bad results for DAOStarFinder.

By the way, that report of mine was written before I knew about this ticket so the data weren't put on GitHub. I just put an example datafile with bad results for DAOStarFinder at /user/goudfroo/temp/JP-2259/jw01088003001_01101_00005_nis_cal.fits.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

Perfect, Paul Goudfrooij, thanks for doing that.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

For reference, here is what I get using the test data from Paul Goudfrooij, which confirms that DAOStarFinder detects lots of Airy ring artifacts as stars, whereas IRAFStarFinder and segmentation.SourceFinder do not.  I'll speak with team coffee to decide what tests end up in the pipeline.

 

!starfind_comparison_niriss_sharphi10.png!

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #8203

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Paul Goudfrooij  or Kevin Volk  this ticket has been set 'ready for testing' by SCSB, so can you let us know the status of this from the NIRISS side, in terms of whether you're able to confirm that NIRISS can sign off on this, after running tests using the improvements implemented by the above PR?

[stscijgbot-jp]

Comment by David Law on JIRA:

Paul Goudfrooij Bumping this ticket again; can you please confirm whether this is now successfully giving you the results that you'd expect from IRAFStarFinder?

[stscijgbot-jp]

Comment by Paul Goudfrooij on JIRA:

Hi David Law - thanks for the prompt. We have already tested using IRAFStarFinder on NIRISS data and we know it is working very well. However, looking at the parameter defaults for tweakreg after this ticket was implemented, I note that DAOStarFinder is still the default for the 'starfinder' parameter for pipeline use, which does not address the concerns in the description of this ticket. Would it be possible to make IRAFStarFinder the default, at least for Instrument = NIRISS? Or should that become a separate ticket?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Paul Goudfrooij Depending on how many instruments/modes prefer to use each type of available star finder, it may be better to make changes via parameter reference files, rather than changing the default in the code. If nearly everyone wants to switch to IRAFStarFinder, then it would make sense to switch the default in the code.

[stscijgbot-jp]

Comment by David Law on JIRA:

Paul Goudfrooij Thanks- in that case I'll close out this ticket since the code itself is working.  I think it would be best/easiest to switch NIRISS to use IRAFStarFinder via a parameter reference file, as that can be done immediately without any need for feedback/testing from the other instruments.  I'll bring up at the next JP meeting for other instruments to consider this though, and we can always swap the code defaults too if everyone's on board.