More accurate estimation of photon/dark current/detector noise on a per image basis

[semaphoreP]

Is your feature request related to a problem? Please describe. From @kjl0025

Just to try to document everything in one place, I'll try to summarize the stuff from Slack here, for the case of having noise on a per frame basis and not in a mean_combine sense as described in above comments:

The photon noise can't really reliably be separated for a single pixel from non-photon noise (like CIC, dark current, read noise, and FPN); dark subtraction subtracts the mean, but the noise is still present. We could include the non-photon noise, though (say, for getting the SNR). However, if a luminous target is being observed, the noise dominating the pixel would be photon noise, of course.

For an estimate of the photon + non-photon noise, we could use sqrt(rn^2 + ENF^2*other_noise^2 + FPN noise contribution) rn: read noise, output from k gain function other_noise: could be estimated as sqrt(pixel value) or could be estimated more accurately from the PTC curve (output from the k gain code), but how we deal with it depends on what EM gain value the PTC was taken at (see below) ENF: extra noise function due to EM gain, defined below FPN contribution: usually estimated to be 0 b/c no good systematic way to deal with it and varies pixel to pixel

The pixel value is really a Gaussian variate (read noise) plus a Poisson variate (dark stuff plus photon signal) plus FPN, but since the mean of read noise is 0 (and since FPN is harder to handle and has roughly 0 noise), it would be approximately correct to assume the pixel value is Poisson (photon + non-photon) noise even though the pixel value is really a variate and not a mean value.

The photon transfer curve (PTC) is one output of the k gain code. I could see the photon noise step function having an input for that (coming from the KGain class). And we could require the input PTC be for EM gain=1 and then multiply the result by the extra noise function (ENF):

def ENF(g, Nem):
    """Returns the extra-noise function (ENF).
    Parameters
    ----------
    g : float
        EM gain.  >= 1.
    Nem : int
        Number of gain register cells.
    Returns
    -------
    ENF : float
        extra-noise function
    """
    return np.sqrt(2*(g-1)*g**(-(Nem+1)/Nem) + 1/g)

where Nem (which is 604) would come from the DetectorParams class, and this function does go to sqrt(2) for large values of g. Or if we had a PTC for some EM gain g1 and knew that the frame in question had g2, we could always multiply by ENF(g2,Nem)/ENF(g1, Nem), although a PTC for unity EM gain would be ideal. And this would be done before dark subtraction for this case since this curve would include contributions due to Poisson dark sources like CIC and dark current (and the FPN does not appear on the linear part of the curve, nor does the read noise).

Originally posted by @kjl0025 in https://github.com/roman-corgi/corgidrp/issues/107#issuecomment-2259302950

Describe the solution you'd like Use the computed read noise as well as PTC from the Kgain calibration step.

Additional context See PR #107 discussion. This would be a much more proper way to address it.

[JuergenSchreiber]

I added the same to #124, but two is better...

[maxwellmb]

@semaphoreP, @JuergenSchreiber , can we close one of either this issues or #124 ?

[kjl0025]

The content of this issue is already verbatim in issue #124 , so we can close this issue.