Non-linearity single/double precision unity/non-unity EM comparison b/w DRP and IIT

[hsergi]

Describe the bug When comparing the non linearity table produced with DRP and the IIT codes, the results agree with different levels of precision. For unity gain frames, the agreement is single precision, whereas for the non-unity gain case, the agreement is double precision.

I inserted the IIT code into the DRP one in calibrate_nonlin.py, so that the version of the libraries used is the same. IIT needs the libraries glob and re, which are not needed in DRP's version. The use of these libraries could potentially change the order of some files. I looked into the file order when the numpy stack of arrays of data is generated:

The file order in DRP and IIT is the same. In DRP, I look into the order in which the stack of frames is generated (*) and it follows the frame ID order (from smaller to greater) for each EM value. In IIT, I look into the order in which its stack is generated and it also follows the frame ID order, as in the DRP.

(*) DRP's input is a Dataset. The Dataset is then split according to CMDGAIN (EM) values. The file order I look at is the one after these operations. That is, the final file order from which each FITS file is open and then the numpy stack of arrays of data is generated.

To Reproduce Switch to branch nonlin_precision_test, load your conda environment (or equivalent) and run under tests/e2e_tests/: python nonlin_e2e.py

The result is:

The differences between DRP and IIT per gain (first unity gain) are: [6.33436068e-08 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] Walker step 3/3: save WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card] Comparing the results with TVAC NL differences wrt ENG/TVAC delivered code (nonlin_table_091224.txt): max=4.0e-05 %, rms=2.8e-06 % Figure saved: l1_to_l2a_output/CGI_EXCAM_L1_0000051731_NonLinearityCalibration.png

Expected behavior We would expect both unity and non-unity gain frames show the same level of agreement. In this case, double precision.

corgidrp version: This was tested with version v1.0

Additional context I checked two more things:

• Mean frame: the mean frame generated by IIT and DRP coincide within double precision.
• CR: including or removing Cosmic Ray rejection in the e2e recipe did not change the results at all.

Final notice regarding the data structure needed to replicate these results: Data from TVAC need to be re-ordered in the same way as in IIT. I did my tests with two folders: one for DRP and one for IIT (with all its sub-folders).

I add the script inspect_tvac.py that I used to recreate the data structure used by IIT when calibrating non-linearity. This script is not part of corgidrp. Do not add it to the main branch when merging any of these results with main.

[semaphoreP]

Is this an important disagreement or is this a "nice to have" improvement in the future?

[nasavbailey]

In my opinion, this is a 'document and use-as-is' issue. Since the L1 data are only 16 bit anyway, the best we'd ever want to do is 2^-16 = 1.5E-5. It looks like we only exceed by a factor of 2-3 and only in the limited case where EMgain=1 and DN<1500. DN = 1500 ~ 15000 photons, or photon noise of ~1% per frame. Seems like to reach a point where this error source becomes non-negligible relative to photon noise, we would need approximately a gazillion frames. I sent this to Eric (wearing his HOWFSC customer hat) and he said that he's not concerned if Bijan is also not concerned. I believe Bijan said in last team meeting he wasn't concerned (I can't seem to tag either of them in this thread, don't know why).

[semaphoreP]

I'm moving this to low priority then.