MIRI imaging calibration

[Guang91]

Hi, I've simulated a grid of bright (but non-saturating) stars using MIRISIM (see below).

Then I derived the encircled energy fraction (EEF, i.e., encircled flux divided by model-input flux) for the simulated sources and compare with the PSF EEF generated by WebbPSF (see below).

The simulated EEF deviates the PSF EEF systematically. I guess this offset is because different CDP files adopted by the pipeline and MIRISIM, is it correct? To account for this systematic, I apply a "gain correction" to all pixel values. The correction factor is calculated as the ratio between the PSF EEF and the simulated EEF at the radius where PSF EEF=80% (the dashed line in the figure above). After correction, the simulated EEF becomes similar as the PSF EEF.

How do you think? Is it the right thing to do? How is the calibration going to be after the launch?

Another issue is that, as you can see from the EEF plot above, the PSF EEF does not reach 1 even at the large radius of 3". This is because the PSF has very extended faint wings. In practice, we will never be able to measure the faint wings due to their low S/N. So, we need another "aperture correction". But the problem is, at what radius should we estimate the correction factor?

[jdavies-st]

Good question. The aperture correction is done in the final source_catalog step (which uses photutils under the hood) of the stage 3 pipeline. The reference file for aperture corrections used is

https://jwst-crds.stsci.edu/browse/jwst_miri_apcorr_0005.fits

Are you seeing this inconsistency in what source_catalog step produces? Or in your own measurements?

Also recall that there's that background double-subtraction issue #4302 you reported which has not been fixed yet.

[Guang91]

Thanks for the response, @jdavies-st. I'm doing my own measurements with TPHOT (http://www.astrodeep.eu/t-phot/), which performs PSF-matched photometry for different bands. The experimental EEF curves above are just calculated with my own little code doing aperture photometry. Is there a way to apply the aperture correction without running source_catalog stage?

Thanks for the update on the double-subtraction issue. Currently, I'm using my own background subtraction with SEP (https://sep.readthedocs.io/en/v1.0.x/index.html), which applies a spatially dependent background estimation (sextractor algorithm). The result looks OK (as the figure above). I'm happy to switch back to the pipeline background subtraction when the issues are fixed.

[stscicrawford]

@Guang91 To comment on your original question: 'The simulated EEF deviates the PSF EEF systematically. I guess this offset is because different CDP files adopted by the pipeline and MIRISIM, is it correct?' This could be it. When running the pipeline, you can specify your own reference files by overriding the parameters. Examples for how to do that can be found here: https://jwst-pipeline.readthedocs.io/en/latest/jwst/introduction.html#running-from-the-command-line

[Guang91]

@Guang91 To comment on your original question: 'The simulated EEF deviates the PSF EEF systematically. I guess this offset is because different CDP files adopted by the pipeline and MIRISIM, is it correct?' This could be it. When running the pipeline, you can specify your own reference files by overriding the parameters. Examples for how to do that can be found here: https://jwst-pipeline.readthedocs.io/en/latest/jwst/introduction.html#running-from-the-command-line

Thanks, @stscicrawford. My mirisim has a folder containing its reference files, e.g., MIRI_FM_MIRIMAGE_AREA_07.00.00.fits MIRI_FM_MIRIMAGE_DISTORTION_07.04.01.fits MIRI_FM_MIRIMAGE_F1000W_PCE_07.00.00.fits MIRI_FM_MIRIMAGE_F1000W_PSF_07.02.00.fits MIRI_FM_MIRIMAGE_F1000W_PSF-OOF_07.00.00.fits MIRI_FM_MIRIMAGE_F1000W_SKYFLAT_07.00.00.fits Is there a way to use these files (instead of the default ones) when running the pipeline? Could you give more detailed instructions? I'm running the pipeline within Python. My current code looks like:

det1 = Detector1Pipeline() result = det1.run(file) which uses the default CRD files.

Also, another thing is I find the pipeline has more CRD files than MIRISIM does. If I just replace part of the pipeline's CRD files (that MIRISIM has), will there be a compatible issue?

Thanks, Guang

[karllark]

I have a feeling that this might be due to different calibration factors being used between MIRISim and the pipeline. By calibration factor, I mean the conversion between DN/s and MJy/sr. @Guang91 : when you analyzed the point source profiles, what pipeline images did you use? Were they the calibration images that have units of MJy/sr?

[Guang91]

I have a feeling that this might be due to different calibration factors being used between MIRISim and the pipeline. By calibration factor, I mean the conversion between DN/s and MJy/sr. @Guang91 : when you analyzed the point source profiles, what pipeline images did you use? Were they the calibration images that have units of MJy/sr?

Yes, they have units of MJy/sr @karllark

[karllark]

Where the units for the MIRISim image you analyzed the same? Just checking.

[Guang91]

Where the units for the MIRISim image you analyzed the same? Just checking.

Just to clarify my procedure: First I run MIRISIM to produce the "pipeline-ready" level 0 files. I then run the JWST pipeline on these MIRISIM-produced files. The pipeline outputs images with units of MJy/sr. I use the output images to produce the EEF curves and compare them with the EEF from WebbPSF (as above).

[jdavies-st]

Is there a way to use these files (instead of the default ones) when running the pipeline? Could you give more detailed instructions? I'm running the pipeline within Python. My current code looks like:

det1 = Detector1Pipeline()
result = det1.run(file)

One can override every reffile type in a pipeline. To find out these overrides:

$ strun jwst.pipeline.Detector1Pipeline --help | grep override
  --steps.dq_init.override_mask 
  --steps.saturation.override_saturation 
  --steps.ipc.override_ipc 
  --steps.superbias.override_superbias 
  --steps.refpix.override_refpix 
  --steps.rscd.override_rscd 
  --steps.linearity.override_linearity 
  --steps.dark_current.override_dark 
  --steps.persistence.override_trapdensity 
  --steps.persistence.override_trappars 
  --steps.persistence.override_persat 
  --steps.jump.override_gain 
  --steps.jump.override_readnoise 
  --steps.ramp_fit.override_readnoise 
  --steps.ramp_fit.override_gain 
  --steps.gain_scale.override_gain 

And to do them given your workflow above:

det1 = Detector1Pipeline()
det1.dq_init.override_mask = "foo.fits"
result = det1.run(file)

[karllark]

For comparing the EEFs between WebbPSF and the pipeline, how do you set the normalizations? I am not sure that it is the calibration factor at all if the comparison is between WebbPSF and the pipeline.

[Guang91]

For comparing the EEFs between WebbPSF and the pipeline, how do you set the normalizations? I am not sure that it is the calibration factor at all if the comparison is between WebbPSF and the pipeline.

For WebbPSF, I didn't normalize the EEF, as I think it has already been normalized such that the EEF->1 as R->∞. For the pipeline image, I normalize the EEF with the model-input flux, that is, the value I told MIRISIM to generate the level 0 files. So, MIRISIM has a role in it.

[karllark]

Ahhh...then it can definitely be the calibration factor.

[Guang91]

Ahhh...then it can definitely be the calibration factor.

Yeah, I agree. Thanks. I'm trying to use the same calibration files in MIRISIM and pipeline. I'll post the results later.

[Guang91]

Is there a way to use these files (instead of the default ones) when running the pipeline? Could you give more detailed instructions? I'm running the pipeline within Python. My current code looks like:

det1 = Detector1Pipeline()
result = det1.run(file)

One can override every reffile type in a pipeline. To find out these overrides:

$ strun jwst.pipeline.Detector1Pipeline --help | grep override
  --steps.dq_init.override_mask 
  --steps.saturation.override_saturation 
  --steps.ipc.override_ipc 
  --steps.superbias.override_superbias 
  --steps.refpix.override_refpix 
  --steps.rscd.override_rscd 
  --steps.linearity.override_linearity 
  --steps.dark_current.override_dark 
  --steps.persistence.override_trapdensity 
  --steps.persistence.override_trappars 
  --steps.persistence.override_persat 
  --steps.jump.override_gain 
  --steps.jump.override_readnoise 
  --steps.ramp_fit.override_readnoise 
  --steps.ramp_fit.override_gain 
  --steps.gain_scale.override_gain 

And to do them given your workflow above:

det1 = Detector1Pipeline()
det1.dq_init.override_mask = "foo.fits"
result = det1.run(file)

I realized it's too complicated to hack into the calibration files. MIRISIM's files are in a format quite different from the pipeline's format. For example, attached is one of the MIRISIM calibration files. MIRI_FM_MIRIMAGE_DISTORTION_07.04.01.zip I cannot see an easy way to implement it to the pipeline. So, I would still use a "scaling correction" to deal with the inconsistencies of calibration files.

[karllark]

I don't know how to use the MIRISim files in the pipeline. As you say, maybe the answer is to just use a scaling correction based on the calibration factors in the two files. It it critical to confirm the origin of the differences you are seeing? Or is just knowing that a difference is possible/likely due to the reference files enough for you to move forward with your work?

[Guang91]

I don't know how to use the MIRISim files in the pipeline. As you say, maybe the answer is to just use a scaling correction based on the calibration factors in the two files. It it critical to confirm the origin of the differences you are seeing? Or is just knowing that a difference is possible/likely due to the reference files enough for you to move forward with your work?

For now, I'm just trying to move forward with my paper.

[Guang91]

@karllark just one final question. After launch, is there a plan to observe some standard stars to re-calibrate MIRI? If so, could you give more details? Then I can add some comments in the paper. Thanks!

[karllark]

Definitely. This will be a major calibration program as part of cycle 1 (and further cycles). Check out https://jwst-docs.stsci.edu/data-processing-and-calibration-files/absolute-flux-calibration.

[Guang91]

Definitely. This will be a major calibration program as part of cycle 1 (and further cycles). Check out https://jwst-docs.stsci.edu/data-processing-and-calibration-files/absolute-flux-calibration.

Thanks for the useful info, @karllark Feel free to close the helpdesk ticket.

[karllark]

Great. Will do.

[karllark]

This issue can be closed.