New data reduction procedure for Spectralon calibration imagery

czender commented 7 years ago

Description

The new hyperspectral imagery calibration procedure (#281) uses dedicated calibration imagery of Spectralon reflectors taken at multiple zenith angles (and possibly one day under both clear and overcast skies). This necessitates changes to the current reduction procedure, and subsequent use of, the Spectralon reflectance imagery.

Completion Criteria

[x] #208 Take Spectralon calibration imagery on clear sky day
[ ] #208 Take Spectralon calibration imagery on cloudy sky day
[x] Convert new spectralon calibration images to netCDF
[ ] Visually inspect then hyperslab images to contain only spectralon panels
[ ] Detect any systematic spatial, spectral, or angular biases in spectralon images
[ ] Reduce (i.e., average) imagery to minimal rank/size that preserves systematic features
[ ] Compute zenith angle for each image time (necessary for xps_wht not xps_drk)
[ ] Annotate reduced images with zenith angles, store in HS extractor directory

smarshall-bmr commented 7 years ago

The VNIR is currently completely out of service. A recent software update to implement x-axis scanning broke the data writing, LemnaTec is working on it now. I have the script ready for the calibration run so I'm hoping it can happen soon.

max-zilla commented 7 years ago

@smarshall-bmr is VNIR back up and running?

ghost commented 7 years ago

Now up and running in predawn mode for calibration

smarshall-bmr commented 7 years ago

VNIR is writing data again. We are working on fixing this 10,000 line scan limit right now, calibration is up next on my priority list.

czender commented 7 years ago

@solmazhajmohammadi @dlebauer The recent calibration images brought to my attention that the SWIR naming convention has diverged from the VNIR. SWIR files now have an extra date string (right after the UUID) in their names, and also are missing an underscore before the english suffix. Was there a good reason for changing the naming convention? It would be helpful if SWIR used the same convention as VNIR. Otherwise we need to re-write the extractors to handle both conventions and be backwards compatible, etc.

zender@cg-gpu01:~/nco$ ls -l /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/2017-04-15/2017-04-15__11-59-12-426
total 481696
-rw-r--r-- 1 dlebauer grp_202     55123 Apr 15 12:03 f130c910-7887-49b0-97bb-db49e8c85e63_2017_04_15_12_05_11frameIndex.txt
-rw-r--r-- 1 dlebauer grp_202     27533 Apr 15 12:03 f130c910-7887-49b0-97bb-db49e8c85e63_2017_04_15_12_05_11image.jpg
-rw-r--r-- 1 dlebauer grp_202 493129728 Apr 15 12:03 f130c910-7887-49b0-97bb-db49e8c85e63_2017_04_15_12_05_11raw
-rw-r--r-- 1 dlebauer grp_202      3503 Apr 15 12:03 f130c910-7887-49b0-97bb-db49e8c85e63_2017_04_15_12_05_11raw.hdr
-rw-r--r-- 1 dlebauer grp_202       869 Apr 15 12:03 f130c910-7887-49b0-97bb-db49e8c85e63_2017_04_15_12_05_11settings.txt
-rw-r--r-- 1 dlebauer grp_202      3561 Apr 15 12:03 f130c910-7887-49b0-97bb-db49e8c85e63_metadata.json

~6 minute discrepancy between SWIR directory and filename stub:
zender@cg-gpu01:~/nco$ ls -l /projects/arpae/terraref/sites/ua-mac/raw_data/VNIR/2017-04-15/2017-04-15__11-56-59-902/
total 5204960
-rw-r--r-- 1 dlebauer grp_202      40591 Apr 15 12:00 ca045a19-7b12-4627-b700-9f51f5829b64_frameIndex.txt
-rw-r--r-- 1 dlebauer grp_202      69299 Apr 15 12:00 ca045a19-7b12-4627-b700-9f51f5829b64_image.jpg
-rw-r--r-- 1 dlebauer grp_202       3605 Apr 15 12:00 ca045a19-7b12-4627-b700-9f51f5829b64_metadata.json
-rw-r--r-- 1 dlebauer grp_202 5329664000 Apr 15 12:02 ca045a19-7b12-4627-b700-9f51f5829b64_raw
-rw-r--r-- 1 dlebauer grp_202      10257 Apr 15 12:01 ca045a19-7b12-4627-b700-9f51f5829b64_raw.hdr
-rw-r--r-- 1 dlebauer grp_202        872 Apr 15 12:02 ca045a19-7b12-4627-b700-9f51f5829b64_settings.txt

czender commented 7 years ago

IMHO the best way to proceed here is to remove the extra date string recently introduced into (some of) the SWIR filenames, i.e., "2017_04_15_12_05_11" in the above example, in all but the metadata.json file. That way the VNIR and SWIR will have the same naming convention. Otherwise we need to re-write the filename handling in the SWIR workflow to handle both conventions. @dlebauer should we wait for a sysadmin to change the names or start to re-write the SWIR filename handling and accept both old and new conventions?

dlebauer commented 7 years ago

Wait on sysadmin. If you send me the appropriate command I can run it. On Mon, Apr 24, 2017 at 5:56 PM Charlie Zender notifications@github.com wrote:

IMHO the best way to proceed here is to remove the extra date string recently introduced into (some of) the SWIR filenames, i.e., "2017_04_15_12_05_11" in the above example, in all but the metadata.json file. That way the VNIR and SWIR will have the same naming convention. Otherwise we need to re-write the filename handling in the SWIR workflow to handle both conventions. @dlebauer https://github.com/dlebauer should we wait for a sysadmin to change the names or start to re-write the SWIR filename handling and accept both old and new conventions?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/terraref/computing-pipeline/issues/289#issuecomment-296844803, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX57V9G7OXzf4-bHQRmI12DUBlHCjDks5rzSiggaJpZM4MYdQC .

czender commented 7 years ago

This should work for a sufficiently priveleged user:

yyyymmdd='2017-04-15'
drc_top="/projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/${yyyymmdd}"
cd ${drc_top}
for drc_sub in `ls -d ${yyyymmdd}*` ; do
   echo "Renaming in directory ${drc_sub}..."
   cd ${drc_top}/${drc_sub}
   for fl in `ls *raw` ; do
       dt_sng=`expr match "${fl}" '.*\([0-9][0-9][0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]\)\.*'`
       uuid_sng=${fl:0:36}
   done # !fl
   echo "uuid=${uuid_sng}, dt=${dt_sng}"
   for sfx in frameIndex.txt image.jpg raw raw.hdr settings.txt ; do
      mv_cmd="/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
      echo "/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
      eval ${mv_cmd}
   done # !sfx
done # !drc_sub

As shown it just does the calibration day. However, all SWIR data since 2016-11-08 appear to use this new convention. @solmazhajmohammadi @LTBen is there a reason for the new convention? If not will Lemnatec please revert to the original (and current VNIR) convention? I do not understand how the hyperspectral extractor ever worked on SWIR files since 2016-11-08. AFAICT it should have reported an error. @max-zilla, I gather that the SWIR extractor was not run this whole time? I hope if it was run that it failed and gave a reasonable error message...

Running the above will suffice to unblock progress on calibration. To rename every SWIR file since 2016-11-08, auto-generate the dates with ls -d 2016-11-0[89] 2016-11-[123]? 2016-12-?? 2017-??-?? from /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR and run the above script on each date...

dlebauer commented 7 years ago

@czender I think everything worked. I added looping over dates (/bin/ls to override alias)

#!/bin/bash

cd /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/
for d in `/bin/ls -d 2016-11-0[89] 2016-11-[123]? 2016-12-?? 2017-??-??`; do 
    yyyymmdd=$d
    drc_top="/projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/${yyyymmdd}"
    cd ${drc_top}
    for drc_sub in `/bin/ls -d ${yyyymmdd}*` ; do
    echo "Renaming in directory ${drc_sub}..."
    cd ${drc_top}/${drc_sub}
    for fl in `/bin/ls *raw` ; do
        dt_sng=`expr match "${fl}" '.*\([0-9][0-9][0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]\)\.*'`
        uuid_sng=${fl:0:36}
    done # !fl
    echo "uuid=${uuid_sng}, dt=${dt_sng}"
    for sfx in frameIndex.txt image.jpg raw raw.hdr settings.txt ; do
        mv_cmd="/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
        echo "/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
        eval ${mv_cmd}
    done # !sfx
    done # !drc_sub
done # !d

Let me know if it looks okay.

dlebauer commented 7 years ago

@LTBen and @markus-radermacher-lemnatec can you please confirm that this will be fixed?

SolmazHajmohammadi-LemnaTec commented 7 years ago

The reason for different naming lays in different headwall software. LemnaTec software is only adding prefix to the file name. I am in touch with headwall costumer support to see if it is an easy fix in the software.

ghost commented 7 years ago

Next step: @max-zilla will run SWIR extractor this week. April 15 is the priority date. @czender will modify the workflow after that.

solmazhajmohammadi commented 7 years ago

@czender @dlebauer it is fixed now

max-zilla commented 7 years ago

@czender I just pulled the latest hyperspectral code and queued 4/15 for VNIR (first) then SWIR. we'll see how things go over the weekend.

ghost commented 7 years ago

@max-zilla - please update

terraref / computing-pipeline

New data reduction procedure for Spectralon calibration imagery #289

Description

Completion Criteria