having trouble simulating an image for a NIRcam observation...

[kdhuga]

c/o Mirage Team:

I've been trying to simulate a simple image for a NIRcam observation but without success. I've attached the script I'm running along with a couple of log files that list the errors that I run into. The log files refer to 2 cases: a) a run with a single yaml testfile, and b) a run with glob read of all yaml files

The main script has been constructed following the directions per miarge/web guidelines; the basic input is taken from the xml and pointing files initially exported from the APT file.

Both runs generate errors in the catalog_seed-image stage of the code.

For option (a) I get the following: File "/opt/anaconda3/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction return ufunc.reduce(obj, axis, dtype, out, **passkwargs) ValueError: zero-size array to reduction operation maximum which has no identity

For option (b) I get the following: File "/opt/anaconda3/lib/python3.9/site-packages/mirage/seed_image/catalog_seed_image.py", line 163, in make_seed used_cats = determine_used_cats(self.params['Inst']['mode'], self.params['simSignals']) KeyError: 'Inst'

Any help to straighten this out would be very much appreciated. best, kalvir dhuga GW astro group dhuga@gwu.edu

Attached files follow: NIRcam-sims.pdf mirage_latest_single_test_f [mirage_latest_glob-read-all_yamlfiles.log](https://github.com/spacetelescope/mirage/files/10358801/mirage_latest_glob-read-all_yamlfiles.log) ile.log

[bhilbert4]

Hmm. It's not obvious to me from the logs why the Inst keyword is missing. Can you send one of your yaml files? If there are details in the yaml that you'd rather not post here, you can also create a new ticket at the JWST Help desk (jwsthelp.stsci.edu). If you select the "JWST General" or "NIRCam" categories, I'll get the ticket.

[kdhuga]

Dear Bryan:

Many thanks for your prompt response --- happy to share the yaml files -- see attached.

cheers, kalvir dhuga

On Fri, Jan 6, 2023 at 5:23 PM Bryan Hilbert @.***> wrote:

Hmm. It's not obvious to me from the logs why the Inst keyword is missing. Can you send one of your yaml files? If there are details in the yaml that you'd rather not post here, you can also create a new ticket at the JWST Help desk (jwsthelp.stsci.edu). If you select the "JWST General" or "NIRCam" categories, I'll get the ticket.

— Reply to this email directly, view it on GitHub https://github.com/spacetelescope/mirage/issues/823#issuecomment-1373705636, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5CM5VS67W3VHEGZ2AYNUKDWRATETANCNFSM6AAAAAATS3SM5I . You are receiving this because you authored the thread.Message ID: @.***>

[bhilbert4]

HI Kalvir, It looks like your attached files didn't make it from your email into the github comment. You might want to try coming to the github issue itself and attaching the files that way.

[kdhuga]

Dear Bryan:

Okay, file attached; yaml-files.zip

kalvir

[bhilbert4]

Ah I think I see the problem. You don't have any source catalogs specified in the yaml files. Sorry, the code should explicitly look for that, but it doesn't. Anyway, you can specify source catalog names when running the yaml generator: https://mirage-data-simulator.readthedocs.io/en/latest/yaml_generator.html#source-catalogs

[bhilbert4]

Or, of course, you can manually add your catalogs to the yaml files.

[kdhuga]

Dear Bryan:

i just tried the following insert in the script (just prior to the yaml generator command:

catalogs = {'NGC5408X-1': {'nircam': {'point_source': ' ptsrc_for_dummy-jet-prop_observations_001.cat', 'galaxy': ' galaxies_for_dummy-jet-prop_observations_001.cat', }, }, }

Unfortunately it generates the same 'Inst' error. Thinking that the inst key is case sensitive I also tried 'NIRCAM', and 'NIRcam' -- all gave the same error.

kalvir

On Sat, Jan 7, 2023 at 7:54 PM Bryan Hilbert @.***> wrote:

Or, of course, you can manually add your catalogs to the yaml files.

— Reply to this email directly, view it on GitHub https://github.com/spacetelescope/mirage/issues/823#issuecomment-1374538899, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5CM5VVJGTKHMIHTFSG53ITWRGNTRANCNFSM6AAAAAATS3SM5I . You are receiving this because you authored the thread.Message ID: @.***>

[bhilbert4]

The error indicates that "Inst" is not in the yaml file at all. Are you using glob to produce a list of yaml files to run? The log makes it look like it's treating observation_list.yaml as one of your exposure yaml files. Could that be it? If you ignore that file does it work?

[kdhuga]

Dear Bryan:

My apologies for dropping off the radar in the last week or so --- life just got in the way. I wanted to let you know that your suggestion of inputting the catalogs by hand was a good one and i did manage in the end to produce some images. That's the good news. The not so good news is that the final images (read from the *linear.fits files) are not quite what I expected. Two things (in my opinion) seem to be out place:

a) way too many what I think are cosmic rays and not real astrophysical sources, and b) the images are identical for all the different targets/exposure settings

Regarding the first point I have tried changing the library (SUNMAX to SUNMIN) and the scale factor (from 1.0 to 0.0) for cosmic rays in the input file but apparently with little effect. Regarding the second point, I have tried running the script for one exposure at a time for two different targets with different catalogs but again the output fields seem to be identical as if somehow the program is generating the background for one fixed catalog for all observations.

At this stage I'm stumped for ideas, any suggestions you can offer would be most appreciated.

Again, many thanks for your previous help; I certainly wouldn't have gotten as far as I have without your input.

cheers, kalvir dhuga ps. I can send you whatever files you think might help to diagnose the problem.

On Sat, Jan 7, 2023 at 1:03 PM Kalvir Dhuga @.***> wrote:

Dear Bryan:

i just tried the following insert in the script (just prior to the yaml generator command:

catalogs = {'NGC5408X-1': {'nircam': {'point_source': ' ptsrc_for_dummy-jet-prop_observations_001.cat', 'galaxy': ' galaxies_for_dummy-jet-prop_observations_001.cat', }, }, }

Unfortunately it generates the same 'Inst' error. Thinking that the inst key is case sensitive I also tried 'NIRCAM', and 'NIRcam' -- all gave the same error.

kalvir

On Sat, Jan 7, 2023 at 7:54 PM Bryan Hilbert @.***> wrote:

Or, of course, you can manually add your catalogs to the yaml files.

— Reply to this email directly, view it on GitHub https://github.com/spacetelescope/mirage/issues/823#issuecomment-1374538899, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5CM5VVJGTKHMIHTFSG53ITWRGNTRANCNFSM6AAAAAATS3SM5I . You are receiving this because you authored the thread.Message ID: @.***>

[bhilbert4]

I haven't updated Mirage to use a cosmic ray rate that matches what's been observed in flight, but I think the rate at the moment is ~reasonable. These should all be removed during the line-fitting step of the JWST calibration pipeline. What readpattern and number of groups are you using for your simulations? In general the NIRCam team recommends keeping individual integrations under about 1000 seconds in order to prevent too many pixels from being affected by cosmic rays.

For the second issue, if you take one yaml file for each target, can you confirm that the catalog entries are correct, and also that the telescope pointing (ra, dec entries) are roughly correct? I'm assuming that your targets are not close enough to each other that the same background sources would be visible in both?

[kdhuga]

Dear Bryan:

Attached are relevant files for two targets i.e, target-1(observation-1) and target-4(observation-7): the files include yaml and point-source catalogs.

TARGET-1: name: NGC5408X-1 # Name of target target_ra: 210.83179 # RA of the target, from APT file. target_dec: -41.38297 # Dec of the target, from APT file. observation_number: '001' # Observation Number Telescope pointing for Target-1: ra: 210.83184814468453 # RA of simulated pointing dec: -41.38299869857642 # Dec of simulated pointing

Target-4_name: HOL-IX-X1 # Name of target target_ra: 149.47217 # RA of the target, from APT file. target_dec: 69.06336 # Dec of the target, from APT file. observation_number: '007' # Observation Number Telescope pointing for Target-4: ra: 149.4722922890577 # RA of simulated pointing dec: 69.06333141093295 # Dec of simulated pointing

Readout pattern same for both observations: Readout = shallow2 (exposure <1000s) groupe =10 integrations=6

I hope this helps.

cheers, kalvir

On Mon, Jan 23, 2023 at 2:12 PM Bryan Hilbert @.***> wrote:

I haven't updated Mirage to use a cosmic ray rate that matches what's been observed in flight, but I think the rate at the moment is ~reasonable. These should all be removed during the line-fitting step of the JWST calibration pipeline. What readpattern and number of groups are you using for your simulations? In general the NIRCam team recommends keeping individual integrations under about 1000 seconds in order to prevent too many pixels from being affected by cosmic rays.

For the second issue, if you take one yaml file for each target, can you confirm that the catalog entries are correct, and also that the telescope pointing (ra, dec entries) are roughly correct? I'm assuming that your targets are not close enough to each other that the same background sources would be visible in both?

— Reply to this email directly, view it on GitHub https://github.com/spacetelescope/mirage/issues/823#issuecomment-1400844158, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5CM5VUS47GKP3WYRF5LLV3WT3JWPANCNFSM6AAAAAATS3SM5I . You are receiving this because you authored the thread.Message ID: @.***>

[bhilbert4]

That all looks reasonable. But you're saying the scenes for these two exposures look identical in terms of the astronomical source on the detector? Is the scene correct for one of the targets?

[kdhuga]

Dear Bryan:

Good point -- will check that out.

kalvir

On Mon, Jan 23, 2023 at 3:50 PM Bryan Hilbert @.***> wrote:

That all looks reasonable. But you're saying the scenes for these two exposures look identical in terms of the astronomical source on the detector? Is the scene correct for one of the targets?

— Reply to this email directly, view it on GitHub https://github.com/spacetelescope/mirage/issues/823#issuecomment-1400960383, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5CM5VRLCSIDBVEAVAQNUZ3WT3VHBANCNFSM6AAAAAATS3SM5I . You are receiving this because you authored the thread.Message ID: @.***>