Data pipeline products and their description

[MateuszKM]

The readthedocs description of the pipeline products needs a bit of work. This is somewhat related to (retired) issues #38 and #39

1) Divide the table into engineering and science data products. Former being used for debugging the data. 2) Include sky spectrum (if subtracted) and object cube in the pipeline output 3) Include a description of all fits file extensions in a file, including physical units and, in the case of flags/masks, an explanation of values. 4) Both 2d and cube image data is given as "/pixel", even though the nature of the pixels changes between the two types. Are fluxes really per pixel in the final product rather than physical units?

[MNBrod]

Extending this to propose modifying nomenclature to better reflect data products:

• Rename uncertainty to variance
• Perhaps using cube instead of icube since the distinction is less critical in this pipeline
• Instead of using single letter suffixes, use full words

[scizen9]

If you use the ftools program fv to examine the pipeline output files, this is what you discover. Each file has four extensions: 0 through 3 or ['Primary', 'MASK', 'UNCERT', 'FLAGS']. Each of these extensions has a separate header and they give the following units:

Primary: (whatever BUNIT is set to) MASK: (None) UNCERT: StdDevUncertainty FLAGS: (None) These are defined by the CCDData class which comes from astropy.nddata.

For the 'int' files: 'int', 'intf', 'intd', 'intk' and 'sky' the bunit says: 'electron' / Pixel units

For the 'icube', 'icubew', and 'icubed' files, the bunit says: 'electron' / Pixel units

For the 'icubes' file, the bunit says: '1e+16 erg / (Angstrom cm2 s)' / Pixel units

So, I recommend that any place where the uncertainty is referred to as 'variance' be changed to refer to it as 'StdDev'.

Answering the question as to if the flux calibrated units should be per pixel or per square arcsecond is not trivial. It involves re-learning the calibration procedure in exquisite detail. I'll attempt to do this (for the umpteenth time) and come to a conclusion.

[scizen9]

This also explains why the flux calibrated 'Uncertainty' is different between the two pipelines. The KCWI_DRP primitive, FluxCalibrate.py treats the uncertainty image like a variance and multiplies is by cal ** 2, when it should be multiplied by just the cal since it is, indeed, StdDev. I will submit a pull request for that calculation in FluxCalibrate.py.

[scizen9]

So the flux units are in erg s-1 cm-2 Ang-1, according to the headers of the flux standard files. Our procedure attempts to gather all the light for the standard by summing over a large aperture. Since these are stars having no real extent, there is no areal sky unit in the calibration. Not exactly sure how to convert to an areal unit (arcsec-1), but it begs the question: what is the calibration for extended sources?

[scizen9]

I also want to record that we need to decide how to handle the sky products before we rename 'icube'. Specifically, if we decide to make a separate cube for sky, it should be called 'scube' and 'icube' maybe should remain. Alternately, we could add the sky cube as a separate extension, although, it should have it's own uncertainty, mask, and flag extensions, so I'm not sure that is a good idea.

Also, current nomenclature dictates that the object cube 'ocube' is the un-sky-subtracted frame (specifically in a nod-and-shuffle observation), so this would preclude using 'ocube' for an ordinary, sky-subtracted science cube.

All this is to say that we need to think of all facets before we change any nomenclature.