PIU dimension propagation issue

[oceancolorcoder]

[Transfer from issue #190]

Multi-level: Raw (BIN) ---> L2(HDF5) Also a new output directory to make sure.

This is the full error message:

Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc Traceback (most recent call last): File "C:\Users\fecgi\Documents\HyperCP\Main.py", line 546, in multi2Clicked self.processMulti(2) File "C:\Users\fecgi\Documents\HyperCP\Main.py", line 540, in processMulti Controller.processFilesMultiLevel(self.outputDirectory,fileNames, calibrationMap, flag_Trios) File "C:\Users\fecgi\Documents\HyperCP\Source\Controller.py", line 815, in processFilesMultiLevel Controller.processSingleLevel(pathOut, fp, calibrationMap, 'L2', flag_Trios) File "C:\Users\fecgi\Documents\HyperCP\Source\Controller.py", line 736, in processSingleLevel root = Controller.processL2(root,outFilePath) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\Documents\HyperCP\Source\Controller.py", line 449, in processL2 node = ProcessL2.processL2(root,station) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\Documents\HyperCP\Source\ProcessL2.py", line 2302, in processL2 if not ProcessL2.stationsEnsemblesReflectance(node, root,station): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\Documents\HyperCP\Source\ProcessL2.py", line 2193, in stationsEnsemblesReflectance if not ProcessL2.ensemblesReflectance(node, sasGroup, referenceGroup, ancGroup, uncGroup, esRawGroup, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\Documents\HyperCP\Source\ProcessL2.py", line 1741, in ensemblesReflectance xSlice.update(instrument.Factory(node, uncGroup, stats)) # update the xSlice dict with uncertianties and samples ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\Documents\HyperCP\Source\ProcessInstrumentUncertainties.py", line 189, in Factory es_unc, li_unc, lt_unc = PropagateL1B.propagate_Instrument_Uncertainty(mean_values, uncertainty) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\Documents\HyperCP\Source\Uncertainty_Analysis.py", line 144, in propagate_Instrument_Uncertainty unc = self.MCP.propagate_random(self.instruments, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\anaconda3\envs\hypercp\Lib\site-packages\punpy\mc\mc_propagation.py", line 123, in propagate_random return self.propagate_standard( ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\anaconda3\envs\hypercp\Lib\site-packages\punpy\mc\mc_propagation.py", line 559, in propagate_standard MC_x = self.generate_MC_sample( ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\anaconda3\envs\hypercp\Lib\site-packages\punpy\mc\mc_propagation.py", line 618, in generate_MC_sample MC_data[i] = cm.generate_sample( ^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\anaconda3\envs\hypercp\Lib\site-packages\comet_maths\generate_sample\generate_sample.py", line 100, in generate_sample sample = generate_sample_systematic( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\fecgi\anaconda3\envs\hypercp\Lib\site-packages\comet_maths\generate_sample\generate_sample.py", line 324, in generate_sample_systematic np.dot( ValueError: operands could not be broadcast together with shapes (100,138) (137,)

Originally posted by @fecgiannini in https://github.com/nasa/HyperCP/issues/190#issuecomment-2252960319

[ARamsay17]

The issue is that the calibration files are missing some information:

These lines present in the KORUS file are not present in the .cal files provided:

Thermal Responsivity derived value

THERMAL_RESP NONE '' 0 BU 1 THERM1 -0.01131601 4.95350e-05 -7.488197e-08 4.33976e-11 20.0

This is causing an error in how the straylight class-based error is sliced/interpolated in PIU. I'm working to solve this issue but in a way that is more robust and doesn't create other errors.

[oceancolorcoder]

@ARamsay17 this issue is not KORUS-related. Please refer to discussion in issue #190. Data for @fecgiannini's RENOMOSR cruise can be found on the Box under Data/FICE2024.

[oceancolorcoder]

@ARamsay17 Irregularity in Sea-Bird calibration files should be managed by re-tooling PIU, etc. to utilize the calibrationMap structure already in place rather than re-reading the calibration files (or having to alter these factory files to process).

[fecgiannini]

Thank you for your attention. I hope we can figure it out soon. Let me know if you need further information. Cheers

[oceancolorcoder]

@fecgiannini We have changed how wavebands are gleaned in the uncertainty analysis and also made the interpretation of sparse relative azimuth measurements in ancillary datasets a bit more robust. This should solve the problems with your dataset. However, I only have two of your raw files to test so if you have any more issues, don't hesitate to reach out.

Incidentally, of the two files, only one survived to L2, and it had very large uncertainties mainly deriving from your Li (sky radiance) measurements. The other failed before L2 for the same reason. Here is the Li for RENOMOSR_SR01_1444: which survived, but had large error, e.g.:

And here is the Li for REMOMOSR_SR01_1324, which did not survive past L1BQC:

[oceancolorcoder]

In those plots of Li, the red-dashed spectra are removed as outliers, gray solid spectra are retained, the solid black line is the average, and the black dashed line is the 2-sigma rejection envelope.

[fecgiannini]

@oceancolorcoder Thank you! I "git pull"ed but nothing was updated. Do I have to clone the branch again? Tks

[oceancolorcoder]

@fecgiannini My apologies! Corrections were being tested on our development branch. I have now updated the master branch with these updates. You should be able to pull again and receive these updates.

[fecgiannini]

No problem at all! Thank you