search

terraref / computing-pipeline

Pipeline to Extract Plant Phenotypes from Reference Data
BSD 3-Clause "New" or "Revised" License
21 stars 13 forks source link

Hyperspectral 2018 Level_1 data generation #591

Open max-zilla opened 5 years ago

max-zilla commented 5 years ago

We need Level_1 outputs from 2018 using existing code to recalibrate the new algorithm.

It seems the cst_cnv_trg.nc file has trouble with the new band count - the new algorithm doesn't use this file but we need to use it with existing pipeline on 2018 data then recalibrate based on that output.

Is it possible to avoid relying on this file, or generating an updated version w/ different number of bands?

dlebauer commented 5 years ago

Moving this to a separate comment because it addresses the specific question

Is it possible to ... generate an updated version w/ different number of bands?

Yes!

The NCO command for subsetting the file to the first 939 wavelengths uses the ncks function

ncks -O -d wavelength,0,938 cst_cnv_trg.nc foo.nc

This creates a new file called ‘foo.nc’ that contains only 939 wavelengths. To explain each part:

To check that the output file is of the correct dimensions, dump the meatdata header and print just the dimension defintions:

ncdump -h foo.nc | grep -A 5 dimensions

Dump out the actual values of wavelength along the wavelength dimension: 

ncdump -v wavelength cst_cnv_trg.nc

note that the wavelength variable is both a dimension with integer indices 0:938 (one dimension of the hyperspectral data cube) and a variable with values and units of wavelengths in nm divided into 939 intervals in the range [389,1000].

max-zilla commented 4 years ago

Added the following code: https://github.com/terraref/extractors-hyperspectral/blob/reprocessing2018/hyperspectral/hyperspectral_workflow.sh#L593

# Subset raw NC file bands
    att_sub=${fl_in[$fl_idx]/_raw/_raw_sub.nc}
    att_sub="${att_in}.subset.tmp"
    cmd_subset="ncks -O -d wavelength,0,938 ${att_in} ${att_sub}"
    eval ${cmd_subset}
    if [ $? -ne 0 ] || [ ! -f ${att_sub} ]; then
        printf "${spt_nm}: ERROR Failed to subset raw data bands. Debug this:\n${cmd_subset}\n"
        exit 1
        fi # !err
    att_in=${att_sub}

...however it didn't seem to resolve error:

Terraref hyperspectral data workflow invoked with:
./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw -o /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403/vnir_netcdf_L1_ua-mac_2018-08-18__11-47-47-403.nc
Hyperspectral workflow scripts in directory /home/extractor
NCO version 4.7.9 from directory /home/extractor/anaconda2/bin
Intermediate/temporary files written to directory /tmp
Final output stored in directory /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403
Input #00: /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw
trn(in)  : /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=939,1600,2212 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw /home/extractor/hyperspectral_dummy.nc /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp.subset.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid25.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa workflow_invocation="'./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw -o /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403/vnir_netcdf_L1_ua-mac_2018-08-18__11-47-47-403.nc'" --gaa terraref_hostname=hyperspectral-gn4nd --gaa terraref_version=4.7.9 -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Mon Jul 22 15:30:10 UTC 2019: ./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw -o /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403/vnir_netcdf_L1_ua-mac_2018-08-18__11-47-47-403.nc" /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp.subset.tmp /tmp/terraref_tmp_att.nc.pid25.fl00.tmp
jsn(in)  : /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid25
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw /tmp/terraref_tmp_jsn.nc.pid25.fl00.tmp
ERROR: nco_put_vara() failed to nc_put_vara() variable "cst_cnv_trg_nw"
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_put_vara()
nco_err_exit(): ERROR Error code is -57. Translation into English with nc_strerror(-57) is "NetCDF: Start+count exceeds dimension bound"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

will need someone's help with this.

max-zilla commented 4 years ago

SWIR 2017 error:

./hyperspectral_workflow.sh -d 1 -h --new_clb_mth \
-i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw \
-o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc

Terraref hyperspectral data workflow invoked with:
./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc
Hyperspectral workflow scripts in directory /home/extractor
NCO version 4.7.9 from directory /home/extractor/anaconda2/bin
Intermediate/temporary files written to directory /tmp
Final output stored in directory /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707
Input #00: /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw
trn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=273,384,2726 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw /home/extractor/hyperspectral_dummy.nc /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa workflow_invocation="'./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc'" --gaa terraref_hostname=hyperspectral-2ssvf --gaa terraref_version=4.7.9 -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Mon Jul 15 13:00:47 UTC 2019: ./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc" /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
jsn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid36
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw /tmp/terraref_tmp_jsn.nc.pid36.fl00.tmp
mrg(in)  : /tmp/terraref_tmp_jsn.nc.pid36.fl00.tmp
mrg(out) : /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
ncks -A -C -v wavelength,x,y /tmp/terraref_tmp_jsn.nc.pid36.fl00.tmp /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
clb(in)  : /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
clb(out) : /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp
ncap2 --no_cll_mth -O --hdr_pad=10000 -v -S /home/extractor/hyperspectral_calibration_new.nco /tmp/terraref_tmp_att.nc.pid36.fl00.tmp /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp
ncap2: WARNING prs_cls::ncap_var_init(): Unable to find variable cst_cnv_trg_nw in /tmp/terraref_tmp_att.nc.pid36.fl00.tmp or /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp
nco_dfl_case_nc_type_err(): ERROR switch(nctype) statement fell through to default case, which is illegal, because each type should have a well-defined action. This error may be triggered by using an NCO built with only netCDF3 functionality to examine a netCDF4 dataset that contains a new atomic type (e.g., NC_INT64).
HINT: Configure/build NCO with --enable-netCDF4. Exiting...
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_dfl_case_nc_type_err()
nco_err_exit(): ERROR Error code is 0. This indicates an error occurred in NCO code or in a system call, not in the netCDF layer.
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
hyperspectral_workflow.sh: ERROR Failed to calibrate data in ncap2. Debug this:
ncap2 --no_cll_mth -O  --hdr_pad=10000 -v -S /home/extractor/hyperspectral_calibration_new.nco /tmp/terraref_tmp_att.nc.pid36.fl00.tmp /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp
max-zilla commented 4 years ago

SWIR 2018 error:

./hyperspectral_workflow.sh -d 1 -h --new_clb_mth \
-i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw \
-o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc

Terraref hyperspectral data workflow invoked with:
./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc
Hyperspectral workflow scripts in directory /home/extractor
NCO version 4.7.9 from directory /home/extractor/anaconda2/bin
Intermediate/temporary files written to directory /tmp
Final output stored in directory /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076
Input #00: /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw
trn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=275,384,12268 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw /home/extractor/hyperspectral_dummy.nc /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid23.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa workflow_invocation="'./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc'" --gaa terraref_hostname=hyperspectral-w1jbx --gaa terraref_version=4.7.9 -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Mon Jul 22 18:55:01 UTC 2019: ./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc" /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp /tmp/terraref_tmp_att.nc.pid23.fl00.tmp
jsn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid23
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw /tmp/terraref_tmp_jsn.nc.pid23.fl00.tmp
Traceback (most recent call last):
  File "/home/extractor/hyperspectral_metadata.py", line 780, in <module>
    main()
  File "/home/extractor/hyperspectral_metadata.py", line 776, in main
    testCase.writeToNetCDF(file_input, file_output, " ".join((file_input, file_output)), format, flatten, debug)
  File "/home/extractor/hyperspectral_metadata.py", line 156, in writeToNetCDF
    tempVariable[...] = translate_time(subdata)
  File "/home/extractor/hyperspectral_metadata.py", line 624, in translate_time
    timeSplit  = date(year=timeUnpack.tm_year, month=timeUnpack.tm_mon,
AttributeError: 'NoneType' object has no attribute 'tm_year'
hyperspectral_workflow.sh: ERROR Failed to parse JSON metadata. Debug this:
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw /tmp/terraref_tmp_jsn.nc.pid23.fl00.tmp
max-zilla commented 4 years ago

@Paheding https://github.com/terraref/extractors-hyperspectral/blob/reprocessing2018/hyperspectral/hyperspectral_workflow.sh

max-zilla commented 4 years ago

https://github.com/terraref/extractors-hyperspectral/pull/50/files

new PR here that is still being finalized, result of our first two hackathons. Should be able to support VNIR new camera, SWIR new + old cameras.