Slow processing of L2 uncertainties (top priority)

[oceancolorcoder]

Ideas to pursue:

Reduce the size of Z17 by confining it to protocol geometries (DA).

Build LUTs in lieu of full calculations for given conditions (AR).

[oceancolorcoder]

Pathway discussed Sept 20: calculate partial LUT for min/max parameters (e.g., wind, AOT, etc.) impact on difference between M99 and Z17 rather than calculated Z17 for each spectrum or LUTs for all possible parameterizations.

@ARamsay17

[oceancolorcoder]

pySAS, M99, turning off satellite convolution to Aqua reduces timing from to 2 min.

The snag is in instrument.rrsHyperUNCFACTORY is quite slow, ~160s per ensemble/xSlice:

Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf Process Single Level ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf stationsEnsemblesReflectance Binning datasets to ensemble time interval. Convolving MODIS Aqua (ir)radiances in the slice 67 spectra in slice (ensemble). 7 spectra remaining in slice to average after filtering to lowest 10.0%. Calculating M99 glint correction with complete LUT Calculating Zhang glint correction. Zhang17 Elapsed Time: 4.3 s Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method. warnings.warn( /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method. warnings.warn( Uncertainty Update Elapsed Time: 160.2 s Perform similarity spectrum residual NIR subtraction. offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717

… [Repeat for each ensemble]

Filtering reflectance spectra for negative values.

0.0% of Rrs_HYPER spectra flagged 0.0% of nLw_HYPER spectra flagged Plotting Rrs Plotting Rrs Plotting Es Plotting Li Plotting Lt Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL L2 file produced: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf Output SeaBASS for HDF: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf Writing PDF Report... Level 1A Level 1AQC Adding deglitching plots... Level 1B Adding interpolation plots... Level 1BQC Adding spectral filter plots Level 2 Adding radiometry plots Adding ocean color product plots processFilesSingleLevel, single file - DONE Time elapsed: 14 minutes

[oceancolorcoder]

Without Aqua convolution takes < 1 minute:

Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf Process Single Level ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf stationsEnsemblesReflectance Binning datasets to ensemble time interval. 67 spectra in slice (ensemble). 7 spectra remaining in slice to average after filtering to lowest 10.0%. Calculating M99 glint correction with complete LUT Calculating Zhang glint correction. Zhang17 Elapsed Time: 4.2 s Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method. warnings.warn( /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method. warnings.warn( Uncertainty Update Elapsed Time: 0.3 s Perform similarity spectrum residual NIR subtraction. offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717 80 spectra in slice (ensemble). 8 spectra remaining in slice to average after filtering to lowest 10.0%. Calculating M99 glint correction with complete LUT Calculating Zhang glint correction. Zhang17 Elapsed Time: 4.1 s Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method. warnings.warn( /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method. warnings.warn( Uncertainty Update Elapsed Time: 0.4 s Perform similarity spectrum residual NIR subtraction. offset(rrs) = -9.679655711474021e-06; offset(nLw) = -0.033131963419616335 78 spectra in slice (ensemble). 8 spectra remaining in slice to average after filtering to lowest 10.0%. Calculating M99 glint correction with complete LUT Calculating Zhang glint correction. Zhang17 Elapsed Time: 4.1 s Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method. warnings.warn( /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method. warnings.warn( Uncertainty Update Elapsed Time: 0.3 s Perform similarity spectrum residual NIR subtraction. offset(rrs) = -2.845188536517354e-05; offset(nLw) = -0.034560316042568845 76 spectra in slice (ensemble). 8 spectra remaining in slice to average after filtering to lowest 10.0%. Calculating M99 glint correction with complete LUT Calculating Zhang glint correction. Zhang17 Elapsed Time: 4.2 s Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method. warnings.warn( /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method. warnings.warn( Uncertainty Update Elapsed Time: 0.4 s Perform similarity spectrum residual NIR subtraction. offset(rrs) = 1.2613785225814207e-06; offset(nLw) = -0.030815234070160086 49 spectra in slice (ensemble). 5 spectra remaining in slice to average after filtering to lowest 10.0%. Calculating M99 glint correction with complete LUT Calculating Zhang glint correction. Zhang17 Elapsed Time: 4.2 s Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method. warnings.warn( /Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method. warnings.warn( Uncertainty Update Elapsed Time: 0.4 s Perform similarity spectrum residual NIR subtraction. offset(rrs) = 8.215846949638485e-05; offset(nLw) = -0.020145435286546286 Filtering reflectance spectra for negative values. 0.0% of Rrs_HYPER spectra flagged 0.0% of nLw_HYPER spectra flagged Plotting Rrs Plotting Rrs Plotting Es Plotting Li Plotting Lt Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL L2 file produced: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf Output SeaBASS for HDF: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf Writing PDF Report... Level 1A Level 1AQC Adding deglitching plots... Level 1B Adding interpolation plots... Level 1BQC Adding spectral filter plots Level 2 Adding radiometry plots Adding ocean color product plots processFilesSingleLevel, single file - DONE Time elapsed: 1 minutes

[oceancolorcoder]

Also, while Z17 is apparently not the logjam here, I have so far found that updating Z17 has only increased its size, considering the newly added AOD to 0.5. From 2.47 GB, it goes to 3.28 GB. I can bring it back down to 2.26 GB by truncating AOD of 0 and SZA of 0 (i.e., starting at 0.05 and 10, respectively). Does not seem worth it yet, but will continue to experiment.

[oceancolorcoder]

Two issues to address: Z17 uncertainties at L2 are too slow (i.e., processing time is nearly. as long as data collection time). For this we need a LUT for M99 - Z17.

The second issue is that the convolutions to satellite bands are excessively slow, making even M99 processing interminable. Setting created to forgo convolution uncertainties until a resolution can be identified.

[oceancolorcoder]

One confusing element about the slowness of running Z17 uncertainties is that when zhangWrapper runs Z17 for uncertainty estimates, each run takes much longer than the run of Z17 to do the glint correction:

... 6 spectra in slice (ensemble). Wavelengths extend beyond model limits. Truncating to 350 - 1000 nm. Calculating Zhang glint correction. Zhang17 Elapsed Time: 9.1 s Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.5 SST: 13.1 SSS: 35.6 Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 135.0 zhangWrapper Z17 Elapsed Time: 35.8 s Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.7 SST: 15.7 SSS: 34.8 Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 133.5 zhangWrapper Z17 Elapsed Time: 35.5 s ...