Update hyperspectral to support new VNIR camera bands

terraref / computing-pipeline

Pipeline to Extract Plant Phenotypes from Reference Data

BSD 3-Clause "New" or "Revised" License

21 stars 13 forks source link

Update hyperspectral to support new VNIR camera bands #576

Closed max-zilla closed 4 years ago

max-zilla commented 5 years ago

Between 2017-2018, VNIR camera was replaced. New camera has 939 bands instead of 955, and this is breaking the extractor code for currently unknown reason. May be related to incompatible calibration file or template designed for older camera.

hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw -o /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403/vnir_netcdf_L1_ua-mac_2018-08-18__11-47-47-403.nc

nco_err_exit(): ERROR Error code is -57. Translation into English with nc_strerror(-57) is "NetCDF: Start+count exceeds dimension bound"
ERROR: nco_put_vara() failed to nc_put_vara() variable "cst_cnv_trg_nw"

Will try to update more on error message later

Paheding commented 5 years ago

@remotesensinglab and I noticed this issue during the development of the new calibration protocol. I opened the cst_cnt_trg.nc files, it is indeed containing 955 bands (389-1000nm) since it was developed for old VNIR data. See the screenshot below:

The easiest way, but may not a good one, to fix this issue could be - keep the dimension of variables to 939 by discarding 17 values. Note the new VNIR contains 939 spectral bands ranging from 399 - 1000nm.

dlebauer commented 5 years ago

it would only make sense to discard the 17 values if these are additional, and the other 939 are the same wavelengths as the old sensor

Paheding commented 5 years ago

The new VNIR sensor has 939 bands with the wavelength ranging from 399.98 nm ~ 1000.62 nm, and the old one has 955 bands ranging from 389.41 nm ~ 1000.25 nm.

max-zilla commented 5 years ago

@Paheding david mentioned you had provided an update about this on Monday, is there any code/pullrequest/summary you can provide here? thanks.

max-zilla commented 4 years ago

The hackathons resulted in changes I implemented here: https://github.com/terraref/extractors-hyperspectral/pull/50

Support VNIR new camera bands, as well as SWIR. Required generating some 939-band versions of some calibration files for VNIR, and avoiding the affected code for SWIR (which has no calibration).

Patrick and team have calibration code we can run on new VNIR data once it's generated, and SWIR is also queued.

Paheding commented 4 years ago

@max-zilla Great! Does this new update incorporate the shell script "if statement" we sent last time, as well as environmental loggar issues included in the netCDF? Thanks.