jdavies-st
commented
1 year ago So uncertainties are propagated for per-pixel readnoise and for shot noise (poisson) in calwebb_detector1. Sources of noise that are not specifically propagated are:
- reference pixels - a pedestal is computed and subtracted from the raw data based on amplifier and row(s). 1/f noise would also fall in here I think, but of course this is temporal. This subtracted pedestal has an uncertainty, but it is not propagated.
- dark current - per-pixel zero-frame dark current is subtracted. Again, there is an uncertainty here, but it is not propagated.
- superbias - per-pixel bias is subtracted. Again, there is an uncertainty here, but it is not propagated.
- nonlinearity - there's an uncertainty in the per-pixel non-linearity correction coefficients, but it is not propagated.
- IPC - interpixel capacitance means that adjacent pixels (star pattern) see each other's flux. This is corrected by convolving with the inverse IPC kernel, but this then correlates the noise of adjacent pixels as well. This is not handled.
The poisson variance is computed based on the slope (signal), gain and number of groups.
And the readnoise variance is computed based on the readnoise per read and the number of reads.
Neither explicitly account for anything listed above, and I don't know the levels.
And it definitely does not take into account IPC, as it is correlated across pixels once corrected by the IPC kernel. I.e. the signal in a pixel is not just temporally dependent, but also spatially dependent - it depends on the flux of its neighboring pixels.
Some discussion of this for NIRSpec is here
https://ntrs.nasa.gov/api/citations/20080047752/downloads/20080047752.pdf
And they imply that the true variance is likely ~10% higher due to this IPC correlation once corrected, see eq 9.
There is some more discussion of IPC in these 2k IR detectors here
https://arxiv.org/abs/2209.01831
relating the Euclid, whose detectors are almost identical to those on JWST.
karllark
jhennawi
mwregan2
hbushouse
[Note: this issue has also been raised to the helpdesk. I'm bringing it up here as well to get as many viewpoints as possible]
When @jhennawi and I were first looking at the pipeline-produced level 3 NIRSpec files, we noticed that the quoted uncertainties were a bit low compared to the actual variation in the data. In tracing this backwards, we noticed that this is present already in level 1. Namely, the quoted uncertainties in the rate files do not seem to explain the pixel-to-pixel variations. In the attached plot, I'm showing data from the area of one of the fixed slits (which does not contain a target). I'm taking the difference between two exposures at the same nodding position and comparing the difference to the quoted uncertainties in the pixels. If everything is working well, this should be a normal distribution with a sigma of 1. But as you can see, this is not the case and NRS2 even seems to be offset slightly. I've looked a several different datasets and noticed the same effect, also while using several versions of the pipeline (this plot shows the result from the pipeline with the latest CRDS). This is also not unique to the fixed slit region of the detector.
One proposed solution to this is that the pipeline isn't properly accounting for the 1/f noise. While this may be true, the sky background in this region is surely much larger than the 1/f noise and the distributions are still a single gaussian.
Thanks in advance