simulate only detector, e.g. dark, flat

[hugobuddel]

Is it possible to simulate just the detector effects? And in a consistent way?

In particular, I'm experimenting one of the most simple pipelines we have to design for MICADO: the dark current pipeline. My idea is to simulate ~10 raw dark frames with all effects (QE, noise etc) and 1 dark frame with just the dark current (QE=perfect, no other noise). Then median average the raw darks in the proto-pipeline and compare the result against the perfect pipeline.

Quality doesn't really matter at this point, it is more of a proof of concept to get the pipeline design going. To validate that how we create our designs makes sense.

Problems that I've encountered trying to make these simulations:

• Is it possible to simulate no source, not even sky? What I did was create a filter with an almost zero transmission curve. (All zeros triggers a normalization exception.) Is there a smarter way?
• Is seems that FPA_USE_NOISE toggles both dark current and QE noise, is it possible to decouple those?
• Is it possible to generate different raw darks (or raw flats) that have a consistent dark (and/or QE) pattern? That is, such that stacking them will reveal the pattern. The randomness seems too large for that at first glance, but didn't actually try the stacking.
• How does one use FPA_DARK_MEDIAN and FPA_DARK_STDEV? Putting extreme values in there didn't have the effect I expected (extreme values in the image).

This will always create exactly the same image:

simcado.run(
    src=simcado.source.empty_sky(),
    INST_FILTER_TC="closed", # My almost all zeros filter
    ATMO_USE_ATMO_BG="no",
    INST_USE_AO_MIRROR_BG="no",
    FPA_USE_NOISE="yes",
    FPA_READOUT_MEDIAN=0,
    FPA_READOUT_STDEV=0,
    FPA_DARK_MEDIAN=0,
    FPA_DARK_STDEV=0,
    FPA_NOISE_PATH="FPA_noise.fits",
    filename="just_FPA_noise.fits"
)

My expectation was that a very small variation would occur when changing e.g. FPA_DARK_STDEV. But even setting FPA_DARK_STDEV to 0.0001 changes the pattern significantly. And the pattern is different each time, to the extend that it doesn't seem that stacking them would recover the original pattern.

[hugobuddel]

Adding some astronomy to this question, probably superfluous:

• The 'dark' is the signal that is detected when there is no light entering the detector. This should result in an image with zero for each pixel, but it does not due to the dark current. The noise pattern is stable to some extend, and can be detected by taking many exposures in the dark (with exposure time 0) and stacking them.
• The 'flat' is the signal that is detected with a uniform illumination of the detector. This should result in an image with the same value for every pixel, but it does not due to quantum efficiency variations of the pixels (and the dark variations, and noise). This pattern is also stable and can be detected by observing a uniform pattern generated by a calibration lamp.

The bias is the last step in the optical train (ignoring later electric effects like crosstalk). So it is the first step in the data processing and therefore a useful test-case.

[astroleo]

Hoi Hugo, just a comment to your description of flat: note that even a perfectly flat illumination on a perfectly uniform QE=1 detector would not give a spatially flat signal due to e.g. vignetting and other optical imperfections that happen along the optical train between the flat source and the detector (this is the reason why flats are taken separately for each instrument setting, e.g. filters). Groetjes uit Leiden.

[hugobuddel]

Yeah you're right @astroleo, that was a bit of a simplification. The biggest effect would be that the illumination is not uniform to begin with, for which we'll have an illumination correction.

Related, QE maps(/cubes) measured from the ground would be very nice to have (also as input to SimCADO), but it is not clear to me whether we will get those.

It's exactly these complications that makes the dark current an easier calibration procedure to start with. (But even then we need to ignore things like persistence.)

[hugobuddel]

This will always create exactly the same image:

This statement was wrong it seems. I recall being able to reproduce the same image twice, but I was probably mistaken since I can't do it again.

[oczoske]

We have not yet systematically tried to simulate raw calibration frames, but a dark should be possible. The level is set by FPA_DARK_MEDIAN with units of photons/s (should be electrons/s, but we're using gain=1 - something to keep in mind for the future). FPA_DARK_STDEV is not in the code, so is not used (and why should it? it's a Poisson process). For the Source I would try empty_sky and config parametersATMO_USE_ATMO_BG=no and FPA_USE_NOISE=yes. Note that a dark needs a finite exposure time (not 0!). Also, a pure dark in SimCADO without read-out noise won't have any structure. For a flat, there's no illumination pattern in SimCADO (yet?).

[hugobuddel]

Thanks @oczoske. It would be very nice if SimCADO could simulate calibration data at some point. That is, also with the correct headers etc.

There seems to be one other missing piece to get a dark current: FPA_DARK_MEDIAN is only used when read_out_type="non_destructive" in the call to Detector.read_out(). The only way to change the read out type seems to be editing the code, so that is what I did. Is it possible to change the read out mode in some other way?

The images now to indeed seem to have a featureless dark current. This is enough to work with for developing an prototype dark-calibration pipeline. (The images only consist of horizontal lines though, so something seems to go wrong, but that might be irrelevant for the moment.)

I don't know about the statistical properties about the dark signal. Poisson seems logical. Perhaps the parameters can be interpreted like this FPA_DARK_MEDIAN is the median dark current value for all the pixels and FPA_DARK_STDEV could then refer to the variations between pixels. The Poisson noise can then be used for variation between multiple readouts of the same pixel. As in, FPA_DARK_MEDIAN and FPA_DARK_STDEV are used to create a featured dark map, which is subsequently sampled as a Poisson distribution. For now this is not necessary though; a featureless dark map is simpler.

[hugobuddel]

This strategy worked. I've successfully simulated raw darks and the median of those recovers the value of FPA_DARK_MEDIAN. Notes:

• The dark image currently indeed scales with OBS_EXPTIME, but it shouldn't right? The dark current is independent of exposure time, since it is the signal when no light is entering the detector. And OBS_EXPTIME * nothing is still nothing. Not sure though, we plan to take darks for different exposure times so there might be some dependence.
• My strategy of creating a light-blocking filter backfired: Source.project_onto_chip() breaks down when the image is all zeros. The function divides by the sum of the image for normalization, so the chip array will be full with nans.
• Had to turn non-linearity off too, with FPA_LINEARITY_CURVE="none". (The dark should not be affected by the non-linearity right? But it probably doesn't matter with realistic dark-current values.)

[oczoske]

The dark signal is proportional to exposure time - thermal electrons are released at a constant rate (sufficiently far below saturation, of course). This is why you have to take darks with the same exposure time as the science images (or scale the darks if you're well within the linear regime). Realistic dark current values with realistic exposure times should not get close to saturating the detector.

[astroleo]

Perhaps a confusion with read-out noise which does not () depend on exposure time. () but it does depend on readout mode. Faster readout means more noise.

[hugobuddel]

Thank you @oczoske and @astroleo that makes sense.

[oczoske]

Incidentally, raw detector frames (including calibration frames and header keywords) are on the post-PDR plan for SimCADO.

[hugobuddel]

Incidentally, raw detector frames (including calibration frames and header keywords) are on the post-PDR plan for SimCADO.

That's a bummer @oczoske, because that makes it harder to use SimCADO data in pipeline prototyping. Thanks for the notification; is that plan described somewhere? Not all is lost though:

1) If it is possible to turn components of the simulation on and off at will, then it is possible to create the pixel values that these calibration products should have. It would probably not require that much work to get to a first order approximation. E.g. adding no-source option for dark, and a uniform-source for flat. Would that be possible before PDR? A next step could be adding structure, which I think would mean deconstructing FPA_noise.fits into dark/flat/illumination/... That might be a bit more work (but is a necessity for correct science images too).

2) We plan to design the imaging pipelines in a machine-readable form that can directly generate mock pipelines. That is, pipelines without the actual calculations. These pipelines should know everything about correct header values, so that should make it easy to add the correct headers to the simulated calibration frames described above. (These mock pipelines can then be used to develop our prototypes algorithms.)

So it would be great if SimCADO could at least generate somewhat sensible pixel values for the MICADO calibration frames, even if the headers etc. are not correct.

[hugobuddel]

Sorry, I read your post-PDR as post-FDR. I consider the present already as post-PDR, because there are no critical/open RIXes for MICADO WP Imaging. What do you consider post-PDR @oczoske?