Hyperspectral workflow fails the metadata step with test dataset

[czender]

Description

The hyperspectral_tst* files at http://dust.ess.uci.edu/tmp provide a minimal image and metadata to test trivial changes to the workflow. However, the workflow fails in the metadata stage. Please investigate and fix.

Details

zender@aerosol:~$ hyperspectral_workflow.sh -d 1 -i ${DATA}/terraref/hyperspectral_tst_raw -o ~/hyperspectral_tst.nc
Terraref hyperspectral data workflow invoked with:
hyperspectral_workflow.sh -d 1 -i /Users/zender/data/terraref/hyperspectral_tst_raw -o /Users/zender/hyperspectral_tst.nc
Hyperspectral workflow scripts in directory /Users/zender/terraref/computing-pipeline/scripts/hyperspectral
NCO version "4.6.4-alpha02" from directory /Users/zender/bin
Intermediate/temporary files written to directory /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T
Final output stored in directory /Users/zender
Input #00: /Users/zender/data/terraref/hyperspectral_tst_raw
trn(in)  : /Users/zender/data/terraref/hyperspectral_tst_raw
trn(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_trn.nc.pid77664.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=955,1,1 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/Users/zender/data/terraref/hyperspectral_tst_raw /Users/zender/terraref/computing-pipeline/scripts/hyperspectral/hyperspectral_dummy.nc /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_trn.nc.pid77664.fl00.tmp
att(in)  : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_trn.nc.pid77664.fl00.tmp
att(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid77664.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa terraref_hostname=aerosol --gaa terraref_version="4.6.4-alpha02" -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Wed Jan 11 08:38:09 PST 2017: hyperspectral_workflow.sh -d 1 -i /Users/zender/data/terraref/hyperspectral_tst_raw -o /Users/zender/hyperspectral_tst.nc" /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_trn.nc.pid77664.fl00.tmp /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid77664.fl00.tmp
jsn(in)  : /Users/zender/data/terraref/hyperspectral_tst_raw
jsn(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_jsn.nc.pid77664
python /Users/zender/terraref/computing-pipeline/scripts/hyperspectral/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /Users/zender/data/terraref/hyperspectral_tst_raw /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_jsn.nc.pid77664.fl00.tmp
One or more important file(s) is(are) missing. Program terminated
environmentlogger_raw.hdr is missing
environmentlogger_metadata.json is missing
environmentlogger_frameIndex.txt is missing
hyperspectral_workflow.sh: ERROR Failed to parse JSON metadata. Debug this:
python /Users/zender/terraref/computing-pipeline/scripts/hyperspectral/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /Users/zender/data/terraref/hyperspectral_tst_raw /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_jsn.nc.pid77664.fl00.tmp

Completion Criteria

• [ ] Now it works for @FlyingWithJerome
• [x] Now it works for @hmb1
• [ ] Now it works for @czender

[czender]

Jerome's initial PR has been merged and now the test fails in a new location. @hmb1 please try to get this to succeed. Problem appears to stem from 1x1 geometry of test image:

zender@aerosol:~/terraref/computing-pipeline/scripts/hyperspectral$ hyperspectral_workflow.sh -d 0 -i ${DATA}/terraref/hyperspectral_tst_raw -o ~/hyperspectral_tst.nc Input #00: /Users/zender/data/terraref/hyperspectral_tst_raw trn(in) : /Users/zender/data/terraref/hyperspectral_tst_raw trn(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_trn.nc.pid44981.fl00.tmp att(in) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_trn.nc.pid44981.fl00.tmp att(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid44981.fl00.tmp jsn(in) : /Users/zender/data/terraref/hyperspectral_tst_raw jsn(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_jsn.nc.pid44981 mrg(in) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_jsn.nc.pid44981.fl00.tmp mrg(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid44981.fl00.tmp mrg(in) : /Users/zender/terraref/computing-pipeline/scripts/hyperspectral/calibration_vnir_45ms.nc mrg(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid44981.fl00.tmp clb(in) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid44981.fl00.tmp clb(out) : /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_clb.nc.pid44981.fl00.tmp ERROR: nco_put_vara() failed to nc_put_vara() variable "rfl_img" nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_put_vara() nco_err_exit(): ERROR Error code is -40. Translation into English with nc_strerror(-40) is "NetCDF: Index exceeds dimension bound" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) hyperspectral_workflow.sh: ERROR Failed to calibrate data in ncap2. Debug this: ncap2 -A --hdr_pad=10000 -s @drc_spt='"/Users/zender/terraref/computing-pipeline/scripts/hyperspectral"' -S /Users/zender/terraref/computing-pipeline/scripts/hyperspectral/hyperspectral_calibration.nco /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid44981.fl00.tmp /var/folders/p6/qrjddnkd0vb7nkv45mzg48pm0000gn/T/terraref_tmp_att.nc.pid44981.fl00.tmp

[hmb1]

Hi Charlie, the y=1,x=1 was causing some spooky behaviour in hyperspectral_calibration.nco. To get round it I changed the line :+1:

from: *rfl_img=rfl_rfr_fct;(xps_img-xps_img_drk)/(xps_img_wht-xps_img_drk)**

to
*rfl_img=(xps_img-xps_img_drk)/(xps_img_wht-xps_img_drk)rfl_rfr_fct;**

can also correct it by casting e.g *rfl_img[wavelength,y,x]==rfl_rfr_fct;(xps_img-xps_img_drk)/(xps_img_wht-xps_img_drk)**

let me know what you wish me to commit

[czender]

Please re-enter all the code above using Github's code marker (the angle brackets). Otherwise the syntax is distorted and the syntax is crucial. The key thing about any change is that is must not increase the memory usage by one iota, e.g., as indicated in the comment in the following:

// [frc] = Reflectance of image (plant reflectance)
// 20160826: Pre-assigning to zero would create additional copy and increase required memory
// Instead use implicit casting, and overwrite attributes propagated from rfl_rfr_fct
//rfl_img[wavelength,y,x]=0.0f;
rfl_img=rfl_rfr_fct*(xps_img-xps_img_drk)/(xps_img_wht-xps_img_drk);
rfl_img@long_name="Reflectance of image";
rfl_img@standard_name="surface_albedo";
rfl_img@units="1";

[hmb1]

OK Well Ill implement the second method - That is swapping the arguments about. rfl_img=(xps_img-xps_img_drk)/(xps_img_wht-xps_img_drk)Xrfl_rfr_fct;

[czender]

Sounds good. Please run the new and old methods side-by-side and verify neither memory usage nor answers change (e.g., due to carrying out division integer rather than float).

[czender]

Also, the "spooky behavior" needs to be addressed :) Is it fixable? Users will not know advance to avoid the arithmetic ordering I picked, and will be just as mystified as I by the failure.

[hmb1]

The spooky behaviour has been address by mods in the ncap2 code