Investigate ASN rules for NIRSpec leakcal and nodded exposures

[stscijgbot-jp]

Issue JP-2013 was created on JIRA by Mark Kyprianou:

For 00675012, investigate the asn rules for the combination of leakcal and nodded exposures.

From Howard:

None of the leakcal exposures got included in any of the lvl2b ASN files (spec2 asn’s) for the science exposures (03102 and 03104) and hence none of the science exposures had the leakcal subtracted during level-2b processing. Off the top of my head I’m guessing this might be due to the fact that there’s a mixture of 2 different kinds of associated exposures included in obs 12:

The leakcal’s

Nodded exposures

 

The ASN rules are already setup to look for the 2-POINT-NOD dither pattern, which is in use here, and use the exposures from the opposite nod position as background exposures, to be subtracted from each science exposure as it goes through the calwebb_spec2 pipeline. The spec2 ASN files that were created for each science exposure do list the appropriate opposite nod exposures as background members. But they don’t include the leakcal exposures. I’m wondering if the ASN rules can only handle one situation at a time (e.g. either background or leakcal, but not both) and somehow the background association won out over the leakcals.

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Also, according to the main list of what goes to level 2b, leakcals (imprint) are supposed to get Level 2b processing, which does not appear to be the case either.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

I do not have permission to copy the data from the directory above. Could someone add me or put the data where I can get it. 

 

cp /ifs/archive/test/jwst/test2/distribution/proprietary/jw00675/jw00675012001 .

cp: /ifs/archive/test/jwst/test2/distribution/proprietary/jw00675/jw00675012001: Permission denied

 

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

I have put two folder under ██████████████████████████████████████████ which contains what I had been working from. Let me know if the necessary files are there.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Jonathan Eisenhamer James Muzerolle Howard Bushouse  I have a few questions to start to get me up-to-speed on asn files.

To try and understand the observation set up I opened APT and looked at obs 12. I have attached a screen shot, but the summary this data is a 2 point-nod and there are science and leakcal observations.

Why are there two pool files attached with this ticket (the cvs files)  ? One to deal with science observations and one to deal with leakcal observations ?

In either pool there are 16 exposures related to obs 12 (that are NRS IFU data)

3102: 4 files, 2 for NRS1 and 2 for NRS2. Not an imprint observation 3104: 4 files, 2 for NRS1 and 2 for NRS2, Not an imprint observation 3106: 4 files, 2 for NRS1 and 2 for NRS2, Imprint 3108: 4 files, 2 for NRS1 and 2 for NRS2, Imprint

To me it seems we have twice as many files as we should. Was this observation repeated ? If not I do not understand why there is 3102 and 3104 or 3106 and 3108.  How do I know which science data and leak cal files should be grouped together ? 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

As to why there are 2 pool files here, it looks like they're just from different processing runs of the same data (one from 5/29 and the other 6/10), so it's not surprising they're identical (or nearly so).

As to why there seem to be twice as many exposures as expected, it's due to the fact that the APT uses Mosaic Properties of 2x1 for each exposure, so you've got 2 mosaic tiles, each with a 2-point-nod dither within them, for a total of 4 images on each of the 2 detectors, hence a total of 8 images for each set (8 images not leakcal and 8 images that are leakcal).

No idea why (scientifically) you'd do a mosaic for a point source, but I have a feeling someone was just checking all available boxes to test the entire parameter space (remember this is pre-flight test data).

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

So before we can attempt to fix whatever deficiencies exist in the current spec2 processing, we need to know from NIRSpec gurus what should happen to observations like these? You've got a 2-point-nod dither pattern, which by default indicates that the 2 nods should be used to do background subtraction from one another. But at the same time you've also got leakcal/imprint exposures that need to be subtracted and the leakcal exposures themselves use a 2-point-nod, so should they be "background" subtracted from one another before they get used to do imprint subtraction?

So let's consider the following set of images (temporarily ignoring the 2 mosaic tiles and only considering 1 detector at a time):

im1:  science nod 1

im2: science nod 2

lk1: leakcal nod 1

lk2: leakcal nod 2

When processing im1, should the order of steps be:

1) im1 - im2 = im1_bsub

2) im1_bsub - (lk1 + lk2)/2

or

1) im1 - (lk1 + lk2)/2 = im1_lsub

2) im1_lsub - im2

or what?

Or is this not even a realistic observing scenario?

James Muzerolle 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Howard Bushouse when there are leakcals per dither, they need to be subtracted from the science image at that dither point only - typically this case is selected because there is an expectation that the leakage signal will be different at different dither positions. So, for your example, it should go like this:

im1_lsub = im1 - lk1

im2_lsub = im2 - lk2

im1_bsub = im1_lsub - im2_lsub

im2_bsub = im2_lsub - im1_lsub

 

I think it would be fine to combine these steps if that would be easier to implement. I.e.,

im1_bsub = (im1 - lk1) - (im2 - lk2), etc.

 

It's a bit trickier when users elect to take a leakcal (no pun intended) at only a single dither position. The assumption in that case is that the leakage signal is the same at all dithers, so the automated processing should apply it like this:

im1_lsub = im1 - lk1

im2_lsub = im2 - lk1, etc.

 

There may be instances where users want leakcals at each dither so that they can average them together for improved S/N, but there's no way for the automated pipeline to parse that distinction, so I recommend sticking with the above (at least until we better understand the leakage signal and performance of the correction as a function of sky characteristics).

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Jonathan Eisenhamer is the information above all you need to know how to update the ASN rules for leakcal observations ?

 

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

That should be sufficient information.

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

James Muzerolle Howard Bushouse 

Another question: In general, when doing nod subtraction, should the "off target" nods from other dithers/mosaic tiles be averaged together before the subtraction? This has been discussed before, but cannot track down the discussion

There is a "real world" example now that I would prefer to use: jw01257:004. NRS IFU with 2-POINT-NOD, leakcal, and mosaic. So, re-framing the above question using this specific example: Should the "off target" nods from the other mosaic tiles be averaged together to create the "background" nod to subtract from a science nod? It seems to me the answer is "No": Nod subtraction should only involve the specific pair at a particular pointing.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

IIRC, I believe the directive and/or decision was to associate background nods only within a given mosaic tile. I can picture the conversation and a table of exposures vs. nod positions in some other ticket, but would probably take a while to find it and I think it was in the context of NIRSpec Fixed-Slit or MOS observations, not IFU. For example, if you had 3-POINT-NOD with a 2x1 mosaic, I believe the desired grouping was: ||Exp No.||Nod Pos.||Mosaic Tile|| |1|1|1| |2|2|1| |3|3|1| |4|1|2| |5|2|2| |6|3|2|

exposure 1 would use 2+3 for bkg-sub, exp 2 would use 1+3, exp 3 would use 1+2, exposure 4 would use 5+6, exp 5 would use 4+6, and exp 6 would use 4+5.  Of course this example does not include the additional complication of leakcal/imprint exposures.

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Understood. Thanks!

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Rule definition

Exposures will be part of an association if the following are true:

• Overall optical path matches
• Pool column patttype matches, either 2-POINT-NOD or 4-POINT-NOD
• Pool column mostilno matches
• Pool column expcount matches. Note that science and related leakcal have different values, but the value will be the self-same for all the science, and self-same for all the leakcals.

Resulting associations will have the following membership:

• one type science
• zero, one or three type background
• zero, one, two, or four type imprint

Calibration

Leakcal subtraction occurs before background subtraction on a per-exposure basis. If there is only one imprint member, this imprint exposure is subtracted from all the science and background exposures.

Otherwise, there will be as many imprint members as there are science plus background members. Which imprint to subtract from which science/background is determined by the matching values of the header keyword PATT_NUM, or DataModel.meta.dither.position_number.

Once leakcal subtraction occurs, then background subtraction occurs, averaging the backgrounds and subtracting from the science-minus-leakcal exposure.

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Jane Morrison I have put a full set of associations in

████████████████████████████████████████████████████

As noted above, in trying to get away from fabricated data, this is a real-world example of the exact use case: jw01257:004. There are 48 spec2 associations: 2 detectors 2 dithers 2 science/leakcal * 6 tiles

Any other interested parties, please review to ensure these are the desired associations.

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

PR#⁠7405 resolves the association side of this issue.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Jonathan Eisenhamer I am looking at  jw01257-o004_spec2_00001_asn.json in the directory you put associations for this ticket. 

This association has 1 science file, 1 background and 2 imprint images. 

I downloaded the data and I see one leak cal is for PATT_NUM=1 and the other is PATT_NUM=2. The Science image is PATT_NUM=1 and the background is PATT_NUM=2.  I will test in the pipeline that the correct leak files are being subtracted from the science/background data

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Howard Bushouse looking at the calwebb_spec2.py  code we have

Steps whose order is the same for all types of input.

        calibrated = self.bkg_subtract(calibrated, members_by_type['background'])

        calibrated = self.imprint_subtract(calibrated, members_by_type['imprint'])

 

I think just switching those lines should do what we want. But I will test to make sure with some 1257 data.  

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Howard Bushouse Jonathan Eisenhamer 

I took one of the associations Jonathan made for the 1257 data - downloaded the uncal files and ran caldetector1 on each file. Then I used the association in calspec2. 

In the calwebb_spec2 pipeline put the imprint step before the background subtraction.

It almost works. For the science data it subtracts the correct imprint image.

Then it does to the background step and subtracts the background image (before the leak cal image for the background has been subtracted).

Someplace we need a looping of the imprint step -

for  science image  subtract imprint

 for background image subtract imprint.  

Suggestions on where this looping should be Since the imprint step in only for NIRSPEC data should we just add the hooks into the imprint step to see if there is a background image and if there is then run the imprint step on that file.

 

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Howard Bushouse Jonathan Eisenhamer Tyler Pauly 

I have rewritten the imprint step to take the background images and subtract the appropriate imprint image. I can return these subtracted images, but I have question on the best way to use these background imprint subtracted images because the background subtraction step reads in background image form a list of filenames.

 In calspec2 here is are the lines of code - calling imprint and then background subtraction step (I have reordered them):

calibrated = self.imprint_subtract(calibrated, members_by_type['background'], members_by_type['imprint'])

calibrated = self.bkg_subtract(calibrated, members_by_type['background'])

I was hoping to not have to change the background step at ALL. In order to do that I think I would need to write out the background imprint subtracted images and then update the member_by_type['background'] name to be this new file. But this is not normally how we do things - we pass updated models around.  If I pass the updated background models around then  I am going to have to update the background subtraction - to take a list of background models rather than a list of background filenames. 

Any suggestions on the best path forward ?

 

 

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Jane Morrison A couple of comments.

Concerning the practice that steps only return the modified input is correct. However, that is only convention, and one could return whatever is needed.

However, it is also convention that a step only operates on one input. The new imprint now has multiple inputs, breaking that convention. This, in turn, forces the output convention to be broken.

To avoid breaking the conventions of one input/one output per step, it may be more appropriate that, at the Spec2Pipeline level, there is an explicit loop over the backgrounds to perform the imprint subtract on each background. The inputs to imprint would then be just the "input" and "list of imprints". If there are more than one imprints, the necessary checking on dither position can be done to match which imprint from the list to apply.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Jonathan Eisenhamer 

I am still not sure how to handle the fact that the background subtraction step is reading the list of background from a list of files not  passed models.  So unless I print out the imprint subtracted data or change background subtraction to accept a list of models (that I have subtracted the imprint) I am still stuck.

I could pass the list of imprint images to the background step in the module average_background - before it averages the list of background it first subtracts the imprints.  Then in the imprint step I just work on the science image and the background step which has a list of background is still working on a list background and if a list of imprints exists it subtracts those first.

Does that seem reasonable ?

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

The commentary in BackgroundStep is wrong. The bkg_list argument can be either filenames or models. The datamodels.open function, when given a model, returns the model. So, you should be good to simply pass a list of the imprinted-subtracted background models.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Jonathan Eisenhamer Ah  I did not realize that thanks. 

 

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

In the calspec2 pipeline there is a check if there are more than 1 imprint image and if there is the code only uses the first imprint image.

Here is the line: |if not self.imprint_subtract.skip:| ||if len(imprint) > 0 and (exp_type in ['NRS_MSASPEC', 'NRS_IFU']| ||or is_nrs_ifu_flatlamp(input)):| ||if len(imprint) > 1:| ||self.log.warning('Wrong number of imprint members')| ||members_by_type['imprint'] = imprint[0]|

So I need to remove that now for IFU nodding - but I wanted to check if that is still the case for

NRS_MSASPEC data ?

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Showing my ignorance, but doesn't the imprint image only affect IFU modes? If one is using a MOS mode, one is already, and purposefully, looking at the image produced by the MSA? Not sure why NRS_MSASPEC is included in the code.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Jonathan Eisenhamer imprint subtraction can be applied to both IFU and MOS exposures. For MOS mode, it's analogous to subtracting dark or background that you don't want in the image, but in this case it's leakage through the not completely opaque shutters that are closed.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Jane Morrison check with James Muzerolle about whether there can be more than 1 imprint exposure associated with MOS exposures. I suppose it might be possible if the observation uses something like a 3-point-nod dither pattern and there's an imprint exposure obtained at each nod position. But at the same time I'd also guess that in that case you'd only associate the 1 imprint exposure with the 1 science exposure at each nod position.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle  I am updating calspec2 to using the leakcal exposures for nodded IFU data.  There is a check in calspec2 that an association will only have a single leak cal observation. That of course is not the case now for the IFU nodded data because the science image has a leakcal and if there is a background it cal also have a leak cal image. 

My question is can MSA data have an association that would contain a leakcal observation for the science data and a seperate  leak cal observation for a background image ?

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison currently, we don't allow users to obtain leakcals with MOS observations. I think we wanted to leave in a hook in case we decide such a capability is needed. Regardless, separate background observations (I presume you mean something analogous to an IFU off-source exposure?) are not done with MOS, so that won't be an issue.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

 

 Jonathan Eisenhamer James Muzerolle 

Two more questions.

First question: On Dec 19 Jonathan added this information.

Leakcal subtraction occurs before background subtraction on a per-exposure basis. If there is only one imprint member, this imprint exposure is subtracted from all the science and background exposures.

I am matching the position number of the imprint (leakcal) with the background or science image. I am assuming that whenever we subtract a leak cal image it MUST have the same position number as the science of background image. But the above statement has me wondering about the case were there is only 1 leakcal observation and we have background. If there is only 1 leakcal observation should I not check the position number and just use it for both science and background ?

Second question: On Oc 30 Jonathan add this information:

Also, according to the main list of what goes to level 2b, leakcals (imprint) are supposed to get Level 2b processing, which does not appear to be the case either. Right now leak cal observations are going through calwebb_detector 1 - but no other processing is done on them. Is this correct ?

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Awaiting ultimate answer, but as I understand, when there is only one leakcal, that gets applied to all the science/backgrounds.

The leakcals/imprint do currently go to 2b, as per the rules.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison yes, when there is only one leakcal it should be subtracted from all science & background exposures in the visit (the premise being that the leakage signal is not expected to vary strongly with position in the FOV in that case).

The current processing of leakcals is correct - they should only go through calwebb_detector1 as they are meant to be subtracted from science exposures prior to any other calibration being applied.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Actually, given Jonathan Eisenhamer's comment, I'm confused - are leakcals only processed through stage 1? If not, are the rate files still used for the imprint step?

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle The imprint step is part of the calspec2 pipeline. The imprint rate files are subtracted from the science or background rate files. 

That is correct right ?

 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison yes, that's correct. Sorry, maybe that was a stupid question, I just panicked at bit at the implication that the leakcals were being processed through the entirety of stage 2.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #7426

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Note that the initial fix included in #7426 turned out to be only a partial solution to the problem. Additional fixes were applied in #7440 (JP-3081) and #7438 (JP-3079) that completely fixed the problem. All of which are included in the DMS B9.1.1 patch release.

[stscijgbot-jp]

Comment by Leonardo Ubeda on JIRA:

I tested the pipeline using data from jw01257:004 as a real world example. I think that the associations are created correctly and the pipeline is applying the IMPRINT correction in the order that is expected, so from my end this ticket can be closed.

I used associations jw01257-o004_20231103t183726_spec2_00075_asn.json and jw01257-o004_20231103t183726_spec2_00081_asn.json for my tests.