MARCI Calibration data: ISIS vs PDS discrepancy

[ladoramkershner]

ISIS version(s) affected: noticed during isis3.9.0

Description
I noticed that the most recent version of the MARCI calibration data in PDS archives do not match the most recent version of the calibration files in the ISIS data area (/usgs/cpkgs/isis3/data/mro/calibration/marci).

The PDS archive calibration files are given in ddd format (custom file format from the manufacturers of MARCI) and the ISIS calibration files are cubes derived from these ddd files. When looking at the ISIS calibration cube history (using cathist), they were derived from data with the naming format of ‘dflat_band1_v1.6.ddd’. This name does not match any PDS archive flat files and the naming convention is different (in PDS they are named, for example, ‘vis1flat.ddd’).

Additionally, when I ran the latest PDS calibration flat file through the same process described in the ISIS calibration cubes history, a cube with very similar (if not exactly the same) artifacts, but very different DN’s, was produced.

How to reproduce
Grabbing the ddd file /usgs/cpkgs/isis3/data/mro/calibration/marci/vis1flat.ddd and running it through the following process will reproduce the cube shown on the right side of the image below.

ddd2isis FROM = /home/ladoramkershner/vis1flat.ddd \
              TO   = /home/ladoramkershner/vis1flat.cub

fx F1       = /home/ladoramkershner/vis1flat.cub \
    TO       = /home/ladoramkershner/vis1flat_band1.cub \
    EQUATION = f1/10000.0000 \
    MODE     = cubes

reduce FROM         = /home/ladoramkershner/vis1flat_band1.cub \
    TO           = /home/ladoramkershner/vis1flat_band1_summing1.cub \
    ALGORITHM    = average \
    MODE         = scale \
    SSCALE       = 1 \
    LSCALE       = 1 \
    VALIDPER     = 50 \
    VPER_REPLACE = null \

Possible Solution
@dpmayerUSGS, @scsides, and I think that the data used for the current ISIS calibration cubes may have been received through personal communication. So, the flat files in the PDS archive should be processed and added to the ISIS MARCI calibration area.

Additional context
Below is a comparison of an interesting area of the two cubes; the left cube is the calibration file sitting in the ISIS data area now and the right is a cube produced using the same processing methods, but derived for the PDS flat file.

[jessemapel]

I'm curious where the dividing by 10,000 came from. The calibration description on the PDS doesn't mention it.

[ladoramkershner]

Conceptually, I am not sure where that comes from, but it is in all of the ISIS calibration cubes histories (cathist output). I think with some reading and some digging we could figure out why (or why we should take it out), I just wanted to do a 1 to 1 comparison with the current data.

[dpmayerUSGS]

@jessemapel @ladoramkershner The factor of 10,000 might be a unit conversion factor. The calibration description on the PDS doesn't specify the units of solar spectral irradiance, but the numbers in that table suggest milliwatts/meter^2/nanometer or equivalent. It's unclear what the units are for spectral radiance (I) in I = DN / Exposure Time / Summing / Response Coefficient

[ladoramkershner]

@jessemapel The process I used to convert the ddd files into cube files was exactly what was done (in ISIS) to the current calibration cubes. However, this process seems to align very little with the actual calibration process on PDS. I am wondering if the previous person who processed the data used a custom script to manipulate the data after ingestion and simply did not document/save the script.

[dpmayerUSGS]

@ladoramkershner According to marcical.txt in the latest volume of MARCI data, the flat files were updated beginning with MARCI volume mrom_0033, although there's no description of what changes were made or why.

Other than a possible scale factor issue, can you reproduce any of the cubes in/usgs/cpkgs/isis3/data/mro/calibration/marci/ if you start from .ddd files in https://pds-imaging.jpl.nasa.gov/data/mro/mars_reconnaissance_orbiter/marci/mrom_0032/calib/ or earlier? If yes, that would imply that the flatfiles ISIS is using are simply based on the older versions archived in the PDS.

[dpmayerUSGS]

Via email from Kim Supulver at MSSS:

the updated flatfiles and UV coefficients on volume 33 [in the PDS archive] should be applied to the entire MARCI dataset, volumes 1 to present.

The flatfiles in the PDS archive (vol 33) should be used for ISIS [e.g. marcical], since those are the most recent/updated official deliverables.

[dpmayerUSGS]

I think we can close this because we've received an answer from MSSS to the immediate question of which flatfiles are canonical. I've created #3690 to address the issue of getting the canonical files into ISIS.