Create recipe template for generating a K-Gain Calibration file

[maxwellmb]

Is your feature request related to a problem? Please describe.

No recipe template exists for creating a K-Gain calibration file from L1 data.

Describe the solution you'd like

A recipe template needs to be written to generate a K-gain Calibration file from L1 data.

Additional context

In order to test this you will need to get your hands on the appropriate input data from II&T or TVAC and you will quantitatively compare the calibration file to the outputs of the II&T data pipeline. In addition please describe in your own words (~several sentences to a paragraph) what the recipe does and what the inputs and outputs are. Finally, please make sure the appropriate table entry is complete within the Confluence DRP Implementation Document table in Section 2.

Here's a checklist:

• [ ] Recipe Written
• [ ] Test Written and passing
• [ ] Table Entry Complete

[JuergenSchreiber]

-found a bug in calibrate_kgain: OBSTYPE keyword is in pri_hdr (not ext_hdr) -still some hard coded values: min ncal, min and max binwidth -some axis labels are missing in the plots -it doesn't work right now since there is no OBSTYPE "MNFRAME" in the TVAC files (only "SCI"), see #145, that is requested in the function kgain_dataset_2_list. Then an empty list mean_frame_stack is returned, which is not caught and crashes the calibrate_kgain. The sorting of the dataset substacks seems to be not very robust, number of sub stacks must be chosen exactly etc.

[semaphoreP]

Yeah, it looks like OBSTYPE should be in pri_hdr, and we are inconsistent in our step functions so far on which one we assumed it to be in. Good to correct it as we go!

If you think those variables are useful, should be expose them as tunable parameters in the step function arguments?

[JuergenSchreiber]

I just try to collect some things that popped up during coding, testing, that could be improved.

[semaphoreP]

Gotcha. I think it's ok to make a copy of the data with the right OBSTYPE keyword that we are requesting (assume that the request is successful).

[JuergenSchreiber]

I am running into a division by zero error, which comes from frames with a roi with 0 columns. That comes from the kgain_params: 'colroi1': 1199, 'colroi2': 2000 which means 1 column? I wonder why that ever worked.

File "/home/schreiber/workspaceroman/corgidrp/corgidrp/calibrate_kgain.py", line 530, in calibrate_kgain local_mean_arrays = [ptc_bin2(frame[rowroi, colroi], ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/schreiber/workspaceroman/corgidrp/corgidrp/calibrate_kgain.py", line 116, in ptc_bin2 DN_bin = max_DN / tot_bins

ZeroDivisionError: division by zero
I will notify Sergi

[semaphoreP]

I'm not familiar with the code, but I was wondering if you checked whether the shape of your input dataset into the calibrate_kgain function is the same as what the unit tests were testing. Just to sanity check that you are inputting data of the expected dimensions.

[JuergenSchreiber]

Thanks @semaphoreP, you are right, I had to set return_full_frame to true in prescan_biassub. It was false before, so the size of the array was not as expected.

[maxwellmb]

@JuergenSchreiber, in order to deliver this software we need to have the same input data that you're using run through the TVAC pipeline. My understanding is that this was never done with the most up to date pipeline upon which this code is built. Can you please coordinate with @mygouf about having someone run this piepline for comparison.

[maxwellmb]

Also, please post the comparison to the TVAC result, in the best way you see fit to this confluence page as part of the PR. It will help a lot with the software delivery.