Mirage seed image failed for faint objects

[mtakahiro]

When I run the following,

test_yaml_files = 'yamls/new_focus_field_f150w_ghost.yaml'

imseeds = []
pfile = test_yaml_files
print('Running catalog_seed_image for {}'.format(pfile))
cat = catalog_seed_image.Catalog_seed(offline=False)
cat.paramfile = pfile

cat.make_seed()
imseeds.append(cat.seed_file)
print(cat.seed_file)

Mirage crashed and returned this error message;

AttributeError                            Traceback (most recent call last)
<ipython-input-20-16b606c09bf3> in <module>
----> 1 cat.make_seed()
      2 imseeds.append(cat.seed_file)
      3 print(cat.seed_file)

~/anaconda3/envs/mirage/lib/python3.6/site-packages/mirage-1.1.1.dev38+gcfbe662d-py3.6.egg/mirage/seed_image/catalog_seed_image.py in make_seed(self)
    221         if self.params['Telescope']['tracking'].lower() != 'non-sidereal':
    222             print('Creating signal rate image of synthetic inputs.')
--> 223             self.seedimage, self.seed_segmap = self.create_sidereal_image()
    224             outapp = ''
    225

~/anaconda3/envs/mirage/lib/python3.6/site-packages/mirage-1.1.1.dev38+gcfbe662d-py3.6.egg/mirage/seed_image/catalog_seed_image.py in create_sidereal_image(self)
   1553
   1554             # translate the extended source list into an image
-> 1555             extimage, ext_segmap = self.make_extended_source_image(extlist, extstamps)
   1556
   1557             # Multiply by the pixel area map

~/anaconda3/envs/mirage/lib/python3.6/site-packages/mirage-1.1.1.dev38+gcfbe662d-py3.6.egg/mirage/seed_image/catalog_seed_image.py in make_extended_source_image(self, extSources, extStamps)
   3486                 # Make segmentation map
   3487                 indseg = self.seg_from_photutils(stamp[l1:l2, k1:k2] * entry['countrate_e/s'],
-> 3488                                                  entry['index'], noiseval)
   3489                 segmentation.segmap[j1:j2, i1:i2] += indseg
   3490         return extimage, segmentation.segmap

~/anaconda3/envs/mirage/lib/python3.6/site-packages/mirage-1.1.1.dev38+gcfbe662d-py3.6.egg/mirage/seed_image/catalog_seed_image.py in seg_from_photutils(self, image, number, noise)
   3534         # represents the single frame noise for the appropriate
   3535         # observing mode
-> 3536         map = detect_sources(image, noise * 3., 8).data + number
   3537         return map
   3538

AttributeError: 'NoneType' object has no attribute 'data'

I executed the same command but with a catalog with bright objects (while here mag>26), and this did not happen. So I suspect that faint sources may get cut off during some process, and ended up as NoneType?

Is this a bug, or am I using Mirage in a wrong way? Files used here are attached.

Thanks in advance, Takahiro

new_focus_field_f150w_ghost.yaml.txt

jw00042001001_01101_00001_nis2_uncal_pointsources_ghost.txt

[bhilbert4]

Sorry Takahiro, this got lost in the shuffle. I think this is due to the arbitrary input parameters I put in the call to detect_sources. It needs a 2d data array, a threshold value v, and a number of pixels n. A source is then defined as a collection of >= n pixels with signal values >= v. For extended sources (which is where Mirage failed for you), Mirage makes a crude estimate of the noise based on readnoise and background, and then uses 3 x this value for the threshold. The number of pixels is set to 8. I didn't want to try measuring the background in the extended stamp image, since the source in that image can be pretty much anything from a point source to a nebula that fills the array. It sounds like I need to be smarter about how to identify the pixels containing the source. I'm not sure at the moment how to do that and still have it work well for the wide range of possible sources. I'll have to give it some thought.

[mtakahiro]

Thanks, Bryan! Is there any way for a user to tweak these numbers, maybe from a yaml file or something similar, rather than touching the values in the code? No worry if not. I will just leave it for a while, and focus on just bright sources for now.

To make it clear, I'm providing a postage stamp that was taken from an observation. Does making a noiseless postage stamp somehow and providing it to Mirage help the situation?

[bhilbert4]

I don't think a noiseless stamp image would help much. The threshold value that Mirage uses would still be the same. Right now there's no way for the user to set the threshold value. The disperser is the slowest part of Mirage, so I was trying to minimize the number of pixels that are sent to the disperser. Maybe rather than requiring pixels to be 3-sigma above the background, I should just require them to be >= the background. Can you send me your postage stamp? I can play around with various threshold values to see what works well and what doesn't.

[bhilbert4]

Note for future development: it may be worth looking into using detect_threshold() from photutils for this. Seems like it would be helpful in the case of WFSS data where the background can vary significantly across the field of view.

[bhilbert4]

Proposals from @NorPirzkal: He's going to try out some options.

• Rather than using a single threshold across the detector, simply include all pixels with fluxes above (e.g.) 0.1% of the object's peak flux.
• Lower the threshold to 1-simga from the current 3-sigma.
• Surface brightness option: keep all pixels that contribute at least X% of the object's total flux.
• Keep the current set-up, but if no or a very limited number of pixels are above the threshold, then lower the threshold for the object until some reasonable number of pixels are above the threshold.

[bhilbert4]

@mtakahiro I just merged a fix for this. Mirage now has a couple new entries in the yaml file, where the user can set the minimum signal rate value that will go into the segmentation map. I set the default value to be significantly lower than where it was before, so hopefully it will find your dimmer galaxies now. But if that limit is still too high, you can lower it in your yaml files.

After downloading the updated code, you can either:

1. re-create your yaml files by repeating your call to yaml_generator() (See here for the new keywords in the call to yaml_generator: https://mirage-data-simulator.readthedocs.io/en/latest/yaml_generator.html#signal-limits-for-the-segmentation-map)

2. or you can manually add the new entries to your existing yaml files. See the bottom of the simSignals section of the example header for the two new entries: https://mirage-data-simulator.readthedocs.io/en/latest/example_yaml.html#simsignals You'll also have to add the new 'photom' keyword to the Reffiles section. You can just set that to 'crds'.

[mtakahiro]

Thanks so much, @bhilbert4 @NorPirzkal ! I will give it a try as soon as possible, and let you know if there are any questions!