Add detection on downsampling, change default value for max. reflectance

[FlyingWithJerome]

Now the hyperspectral workflow can correctly captured the downsampling case by checking scanSpeedInMPerS. Probably we need to change somewhere else in the workflow since the output coordinates of the pixels won't change the look of the images (but the data do change).

The data in November 30th was clearly downsampled, and is one of the testcases:

hyperspectral_workflow.sh -d 1 -i /projects/arpae/terraref/sites/ua-mac/raw_data/VNIR/2016-11-30/2016-11-30__12-49-05-029/a958998d-465f-4715-9574-674680772ebf_raw

Before downsampling process:

After downsampling process:

We can see the coordinates changed (because we want the pixels to be vertically stretched), but the image did not.

The un-downsampled data are not affected in this update

[czender]

Downsampling changes the y coordinate, not the data. Plotting the same data in the same size image will not change the aspect ratio. Plot the data with true aspect ratio (1 m in x takes same screen space as 1 m in y) to see whether image distortion caused by downsampling has improved. You may not be able to control the aspect ratio with Panoply. Learn/use something else?

[remotesensinglab]

Are we constraining the reflectance values between 0 - 0.6 by setting max reflectance 0.6? This may be ok for pure leaf reflectance but we are missing the mixed pixel issues here if we manually set the max ref to 0.6. I may have missed other related discussion on this. Until we finalize the shadow/soil algorithms and new calibration procedures, would it be better to keep the current process?

[czender]

We are not yet masking any data. We are getting ready to, though, and for that purpose we need a default maximum believable reflectance that is conservative in that it should not be exceeded at any pixel at any wavelength. An exceedance indicates the retrieved reflectance is poor or low quality. It is a caution to the user. No data will be discarded (yet). If, in your opinion, 0.6 is too low, then pick something else and provide some justification.

[czender]

In other words, we are getting ready to create a quality flag, we are not yet changing or constraining the data.

[FlyingWithJerome]

@czender My wording issue data --> coordinates Yes, Panoply might be the problem, since I did some extreme test (e.g., twice the y pixel) and the image was still the same. I will do a little research on finding another visualization tool.

MAXIMUM_PLANT_REFLECTANCE and test on SATURATED_EXPOSURE will be finished very soon.

[FlyingWithJerome]

@czender

This is a better image after our downsampling processing:

With set height and width size ratio both to 100 in Panoply

[czender]

please show the same image without the downsampling correction, so we can compare, and fix it so we can read the y axis labels.

[solmazhajmohammadi]

@czender Any value above 57351 was saturated. As of today, the maximum value is changed to: 2^16 - 1

@FlyingWithJerome I am a little bit confused with your plots. What you mean by "After downsampling process:" the data is already downsampled.

The distance from southeast corner of the field was from 4.6m to 5.3m and after applying the algorithm you get 4.6m to 5.6m It seems to me that you have missed the first and last tile of the image. The image tile (data from 4.6m to 4.8m) in both plot have same coordinate.

[czender]

@solmazhajmohammadi in order to create a propre quality flag or mask for past imagery, it would be helpful to know the date on which the saturated value changed from 57351 to 50000. We know it changed from 50000 to 2^64-1 today. i think your terminology is correct and @FlyingWithJerome has been using the terms as if we were doing the downsampling when in fact we are correct the coordinates for the downsampling that has already been applied. @FlyingWithJerome please implement the default saturated value as 2^64-1 and we will go back later and put in time-varying values when we have that information.

[solmazhajmohammadi]

I am digging into the log files to find the date that has been changed from 57351 to 50000. Just to be clear the maximum exposure is 2^16 - 1 not 2^64 -1

[FlyingWithJerome]

@czender @solmazhajmohammadi @remotesensinglab The default max. reflectance had been set to 60% and the default (and max) saturated exposure had been set to 2^16-1 (65535)

Now the hyperspectral_test.py takes at least 1 and at most 4 command-line arguments. They are:

-bash-4.1$ python hyperspectral_test.py --help
usage: hyperspectral_test.py [-h]
                             input_file_path [verbosity]
                             [maximum_planet_reflectance] [saturated_exposure]

positional arguments:
  input_file_path       The path to the final output
  verbosity             The verbosity of the test report (from 1 <least
                        verbose> to 3 <the most verbose>)
  maximum_planet_reflectance
                        The maximum planet reflectance that has physical
                        meaning (default=0.6)
  saturated_exposure    The maximum saturated exposure that has physical
                        meaning (default=2^16-1)

optional arguments:
  -h, --help            show this help message and exit

And I will keep changing the downsampling related codes based on @solmazhajmohammadi 's feedback

[czender]

The downsampling correction looks good to me. It is at least better than no correction and may well be done accurately. In any case, I prefer to merge this now and continue making improvements later rather than waiting for all issues to be fixed in one pull.

[remotesensinglab]

Yes, looks good to me as well.

[czender]

@FlyingWithJerome please upload, as requested above, new "before and after" pictures of the downsampling correction. We cannot read the labels on the Y axis, and those are crucial to verify that the correction has been done correctly.

[FlyingWithJerome]

@czender I tried to export the picture from Panoply again. Panoply itself crops the coordinates (and I don't know why). The figure below shows the basic idea:

[czender]

Then please email before and after pictures to me and @solmazhajmohammadi Make sure both axes contain labels with sufficient numerical precision to verify that the downsampling procedure works as intended. Then, unless Solmaz objects, I will merge this patch.

[solmazhajmohammadi]

@FlyingWithJerome Thanks, please send it to: Solmaz.hajmohammadi@lemnatec.de

[solmazhajmohammadi]

On 2017-01-17 the maximum exposure has changed from 57351o 2^16-1 sometime between 2017-01-29 to 2017-03-24 the maximum exposure has changed to 50000(we dont have any record on this) but there was no data collection during this time.

[FlyingWithJerome]

@solmazhajmohammadi @czender Right now the max. exposure in the QAQC script is 2^16 - 1 (65535) so it is obviously an expected success testcase and will never fail. Do we need to dynamically change the max. exposure based on the date collected? Just like what you stated above, the max. exposure before 2017-1-7 should be 57351, to make the testcase more meaningful.

[dlebauer]

@FlyingWithJerome @craig-willis should max exposure be stored as part of the metadata that can be queried dynamically rather than hardcoding values and dates in the script?

[FlyingWithJerome]

@dlebauer I also thought about it. I will contact with @solmazhajmohammadi for more information so that the program can dynamically retrieve the max. exposure based on the date recorded.

[czender]

@dlebauer @FlyingWithJerome Yes it would be helpful if the *.raw files held the max exposure so we could just read, preserve, and use it during processing. It's too late to do that for all images to date, so it seems that the max exposure will need to be hardcoded (by date) in the workflow scripts. Or do you mean something else by "queried dynamically"?

[dlebauer]

If it is a relatively stable number (Like different values for before/after Jan 2017) I was thinking a table in the sensor metadata with a date range and value would work. But if this is something the sensor can include in its output that might be better (less chance for error).