Mosaicing around MIRI Lyot looks odd

[stscijgbot-jp]

Issue JP-2164 was created on JIRA by Misty Cracraft:

We've been noticing for some time now that the image combination in calwebb_image3 seems to be doing odd things in the region around the Lyot (and other areas around bad pixel regions and image edges). Several tickets have been raised with this noted, but always combined with other issues. The other issues have been resolved, and now we need to track this problem down. I'll attach a testing notebook in which I have four simulated MIRI images in F770W that are run through the pipeline (all stages) and combined into a mosaic. The flat files mark all of the region between the coronagraphs and the imager portion as DO_NOT_USE (combined with a variety of other flags). The combined image, however, shows some of the image in that section (not on the Imager or Lyot) has negative values in the final mosaic. If this section is made up of four different image portions that are all flagged as DO_NOT_USE, shouldn't that be 0 in the final output, just like the 4QPM regions? In the notebook, you can see (in cell 13) that the portions of the cal image in blue are flagged DO_NOT_USE, but in the final combined mosaic (cell 17), there are portions of the image in between the Lyot and Imager that show up white (negative numbers) rather than blue (0). All of the data from uncal to combined can be found at the box locations shown above.

We need to figure out what's going on with the masking/mosaicing in this region. We've fixed all the other problems that I thought might have helped with this one, made sure the flats had the proper flag masking, and the problem is still persisting.

 

[mcara]

A fix was proposed in https://github.com/spacetelescope/drizzle/pull/79

[nden]

@mcara Please add a change log entry in jwst/CHANGEs.rst pointing to the fix in drizzle.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I've also attached a screenshot of the combined image, shown from -1 to 1 scaling. The black areas near the Lyot are around -3, the 4QPM regions are 0 (as expected), and the sky background between sources is near zero (since the background is currently being subtracted, this is what I would expect to see). It's just the regions around the Lyot and a few other small sections (like the horizontal patches near the bottom left of the image where we have some bad pixels) that don't seem to react as expected.

[stscijgbot-jp]

Comment by James Davies [X] on JIRA:

Maybe the background subtraction should be modulated by the DO_NOT_USE flag? Or the NON_SCIENCE flag? That's my first thought on this, without actually looking at the code.

One of the problems is that both NON_SCIENCE and DO_NOT_USE essentially mean the same thing, and it's not clear why NON_SCIENCE would be set without also setting DO_NOT_USE.

We should only be subtracting background off the areas of the detector that actually see background light.

Is there any case where NON_SCIENCE would be used for any steps in the pipeline level 2 and above?

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I just added an inverted scale F2100W combined image (same field, different filter). I figure the white shows up better and demonstrates the issue more clearly, that the problem isn't just around the Lyot, but image edges, etc.

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Very clear that the issue is widespread around the mosaic. Thanks Misty.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Some initial investigation suggests that this may be simply a matter of not having enough of the pixels with anomalous response flagged as DO_NOT_USE in either the MASK or FLAT ref files. For example, in the F2100W mosaic the bad vertical streaks near the right edge of the mosaic and (somewhat less prominent) horizontal streaks near the bottom, as well as similar "streaking" along the right and bottom edges of the Lyot area, can be completely fixed by just adding the DO_NOT_USE flag to one more column and row at the right and bottom edges of those regions in the cal files. Attached are examples showing the before and after, using those additional flags, for the F2100W mosaic with only the resample step being applied (no tweakreg, skymatch, or outlier_detection). Those prominent streaks are gone in the new version.

Oh, and in addition to the flags around the edges, it appears that the "chicken scratches" in the upper-left of the imager FoV are not flagged at all in the cal files, hence they don't disappear at all in the mosaic. I believe they are flagged in the F700W cal files, which helps to get rid of them. Either the MASK or FLAT ref files for F2100W may need updating to include DO_NOT_USE for those pixels.

I'll continue to investigate what's happening in any of the other steps, like skymatch, to make sure they're all paying attention to the DO_NOT_USE flags.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

!F2100W resample only with standard flags.png|thumbnail!

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

!F2100W resample only with additional flags.png|thumbnail!

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

This is great news. Many thanks for this.

It sounds like you are saying that this is in the flat field file and we need to grow the DO_NOT_USE regions by a pixel or two and/or ensure that all the flat fields have all the bad regions set to DO_NOT_USE. Is this correct? If so that would be great as it is an "easy" solution.

[stscijgbot-jp]

Comment by James Davies [X] on JIRA:

What about the area between the main imaging area and LYOT? It looks like in the images Misty included from June 28 and Sep 17 both show a background level being subtracted in this region, but it clearly sees no sky, but it doesn't seem to be labeled as DO_NOT_USE.

FWIW, the background level subtraction in resample is not sophisticated:

https://github.com/spacetelescope/jwst/blob/066f6962adf4ad215ff62be71d92085f62f817a7/jwst/resample/resample.py#L126-L127

But it doesn't really need to be, as long as areas of the detector that do not see light from the sky are flagged as DO_NOT_USE.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I'm pretty sure the region between the lyot and imager is labeled in the flat field DQ arrays as DO_NOT_USE. If that's not being picked up somewhere, that is a problem. I looked at jwst_miri_flat_0655.fits, which is the F2100W full frame flat, and that region is flagged as 67, which indicates it is properly flagged for DO_NOT_USE.

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

I've confirmed that all of the input cal files have DO_NOT_USE set for the area between the Lyot and Imager, as well as in places like the Lyot spot and its supports. The CON and WHT arrays in the resampled products confirm that no inputs contributed to those pixels (set to zero), but yet the resampled SCI array has the negative of the sky value in those pixels. The areas all around the outside of the Lyot (e.g. to the upper-left and lower-left) are all properly set to zero in the resampled SCI. Only the small portions bridging the gap have non-zero (and in this case negative) values. Will look into the order in which the sky values are subtracted during resampling.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

It sounds like you are saying that this is in the flat field file and we need to grow the DO_NOT_USE regions by a pixel or two and/or ensure that all the flat fields have all the bad regions set to DO_NOT_USE. Is this correct?

You are correct, sir.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

There's something odd going on here, in that all of the pixels in the unilluminated areas between the Imager FoV and the 4QPM and Lyot regions are all flagged with same DQ=262657 (NO_FLAT_FIELD + NON_SCIENCE + DO_NOT_USE) value, which causes all of those pixels to end up with values of zero in both the CON and WHT arrays of the output resampled image (as they should), yet for some reason only the pixel in between the Imager and the Lyot region are having their values get resampled into the output image, thus causing those negative regions (due to sky subtraction). Meanwhile, the regions around the 4QPM's, which are flagged in exactly the same way, come out zero in the resampled image.

So I'm calling in the big guns on this one (James Davies [X]) to try to figure out what's going on inside of the resample step, with the way it's handling the input DQ values, setting up a corresponding mask for use when resampling, building the output WHT array, etc.

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Howard Bushouse Any update?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Not yet. Mihai has been tied up with some other stuff temporarily and James is essentially gone.

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Ok. If there is anything I can do to help, please tell me. This is a blocker in my mind to getting good MIRI imager data right now.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fortunately the only remaining issue at this point is why the pixels in between the Lyot and Imager regions retain their non-zero values when their CON and WHT values are zero. All of the wild streaks and spots throughout the combined image go away when more aggressive DO_NOT_USE flagging is employed in some of the reference files (flat, mask, ...) in order to exclude a few more unreliable pixels around the perimeters of the different regions (Lyot, 4QPM, Imager).

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Just to confirm, do the region inside the image FOV (e.g., 3 parallel "scratches" in the upper left) go away with an update to the flat field reference file (i.e., "more aggressive DO_NOT_USE flagging")? My understanding is that these are already flagged as DO_NOT_USE and yet were being used. If my understanding is correct, then they would be in the same category as the pixels between the Lyot and Imager regions and, hopefully, will be removed when the fix for these pixels is found. The screenshot given still shows some of these issues at least (especially the elliptical region near the lower left).

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Ok. I've been exploring this issue. I agree that the flat field reference file is very likely at fault for the "scratches" and other similar artifacts. I will continue to investigate with a modified flat field to confirm that this solves the issue.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Yes, attached are snaps of one of the science images and the DQ array from the flat ref file for the region with the diagonal scratches. As you can see, the flags in the flat ref file are not always continuous across the entire extent of the scratches or perhaps simply don't go deep enough (e.g. covering pixels that have smaller levels of anomalous signal). !Screen Shot 2021-11-15 at 3.39.20 PM.png|thumbnail! !Screen Shot 2021-11-15 at 3.39.39 PM.png|thumbnail!

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Mihai Cara example data can be found in ███████████████████████████████████████████ The example ASN file "F2100W_image3_asn.json" uses the 4 input CAL (like FLT) images "det_image_seq[1234]_MIRIMAGE_F2100Wexp1_cal.fits" as input to be combined. The entire calwebb_image3 pipeline, which includes tweakreg, sky_match, outlier_detection, and resample, can be run using:

strun calwebb_image3 F2100W_image3_asn.json

which will result in the combined/resampled image "F2100W_combined_i2d.fits". If you want to run the pipeline with some of the steps skipped you can use:

strun calwebb_image3 F2100W_image3_asn.json --steps.tweakreg.skip=true --steps.sky_match.skip==true

If you look at the SCI array in the combined image, you'll see that pixels in the upper-left of the image that are between the coronagraphic mask and the main imager field of view are left with values equal to the negative of the sky level, instead of being zeroed out. For example, the region of pixels [321:349, 756:875] (in FITS 1-indexed and x,y order) have SCI values of -332, even though they are zeroed out in the WHT and CON arrays. They get zeroed out the WHT and CON arrays because they are flagged as DO_NOT_USE in the DQ array of the input CAL files.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed in the drizzle package by spacetelescope/drizzle#79

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Example of new result for F2100W mosaic. The regions around and between the Lyot and main imager are now properly set to zero in the SCI array, based on their zero values in the CON and WHT arrays. Interesting side effect is that the outlier_detection step (using default param values) is flagging a number of pixels in the bright cores of several of the galaxies (probably erroneously), causing them to be rejected during resampling, but in the old drizzle code they were still getting large non-zero values in the SCI array (despite the fact that they're set to zero in CON and WHT arrays), so you didn't notice the bad CR flagging results. The new version correctly sets those pixels to zero in the SCI array, so you DO notice the holes in the nuclei (causing you to go back and either turn off or tweak the params for the outlier_detection step). The example shown here is with outlier_detection skipped, so no holes in the nuclei. !Screen Shot 2021-12-28 at 9.09.11 AM.png|thumbnail!

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Setting this to ready for testing and assigning back to the MIRI team.

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Yeah!

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I re-ran my resample notebook and noticed that the area between the Lyot and the imager now shows zero like it should, and the Lyot region itself (given my F1130W input data), looks smooth. I did not see holes in the middle of the stars as Howard described. I also ran the outlier detection notebook, which uses the same F1130W data, and it gave good results as well. I'll do more tests on the F2100W data to see how changes in parameters change what is removed from the centers of the stars, but for now, I agree that the issue this ticket was filed for has indeed been fixed.

[stscijgbot-jp]

Comment by Mihai Cara on JIRA:

I want to confirm Howard's observation about clipping. I have attached a screenshot from DS9. I agree that it seems the issue from this ticket has been solved and this clipping is a totally different issue and, maybe it is not an issue with software, but rather data and/or settings.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Yes, the "clipping" in the bright galaxy cores is just overly aggressive flagging by the outlier_detection step when using all the default param values for that step. It clearly needs optimized param values to match the MIRI imaging characteristics.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Testing has shown that the issue for which this ticket was opened has been fixed. This can also be seen in multiple pipeline testing notebooks that run through calwebb_image3, such as this source catalog notebook: https://jwst-validation-notebooks.stsci.edu/jwst_validation_notebooks/source_catalog/jwst_source_catalog_miri_test/jwst_source_catalog_miri_test.html