Develop new calibration procedure for Hyperspectral Imagers

[dlebauer]

@czender commented on Thu Mar 02 2017

Description

We discussed pitfalls of current hyperspectral (HS) calibration procedure and agreed to attempt to improve calibration by expanding the temporal/angular resolution of the Spectralon reflectance imagery, utilizing the spectral downwelling irradiance from the Environmental Logger (EL) sensor, and possibly incorporating radiative transfer modeled (RTM) fluxes to extend the effective bandwidth of the EL sensor beyond 1 micron, so the technique can be used for the SWIR imagery.

Details

1. Lemnatec developed the current (someday to be "old") HS calibration procedure, employed since 201609. This procedure has been based on a single set of exposures made 20161021, and does not use any "real time" data from the EL. The script that "reduces" the spectralon exposures to the form used in the HS workflow is here
2. An older HS calibration procedure utilizes "real time" irradiance from the EL. This was never fully implemented due to lack of input data (spectralon imagery in particular, but also spectral irradiance data).
3. The "new" calibration procedure will combine elements from both of the above procedures, and yield reflectance data that better accounts for zenith-angle and cloud effects, which are both large.

Completion Criteria

• [x] #282 Update and document new calibration procedure
• [x] #282 #289 Update new calibration procedure to account for dark counts
• [x] #208 Collect clear sky diurnal hyperspectral spectralon images for calibration
• [ ] #208 Collect overcast sky diurnal hyperspectral spectralon images for calibration
• [ ] #289 Create new reduction procedure based on current reduction here
• [x] Interpolate EL irradiance to time of HS image @FlyingWithJerome
• [ ] Interpolate Spectralon exposure to cosine zenith angle of HS image
• [ ] Modify HS calibration to use interpolated irradiance and spectralon exposure (VNIR+SWIR)
• [ ] Validate/quality control VNIR reflectance for clear sky (Milestone 1)
• [ ] Compute clear sky RTM fluxes for zenith angle of HS image
• [ ] Extend EL-observed spectral irradiance with RTM irradiance (SWIR only)
• [ ] Validate/quality control SWIR reflectance for clear sky (Milestone 2)
• [ ] Obtain or diagnose cloud fraction
• [ ] Compute all sky (with clouds if present) RTM fluxes for zenith angle of HS image
• [ ] Use interpolated cloudy sky spectralon exposure, RTM fluxes in HS calibration
• [ ] Validate/quality control VNIR+SWIR reflectance for all sky (Milestone 3)

• [ ] Shadow detection algorithm/correction?

@czender commented on Thu Mar 02 2017

As y'all can see, I have created this issue as an "epic" for the new calibration procedure. Please comment add modify. In the next few days I'll start to add some sub-issue that correspond to the completion criteria. If you know what needs to be done, please feel free to add those sub-issues yourself and link back to this epic.

[smarshall-bmr]

At the moment I'm still working with LemnaTec trying to get the SWIR and VNIR to take full data stripes. I'll get to this as soon as possible but I'm working through issues one at a time.

[dlebauer]

Do the sensors need to take full data stripes to generate the calibration images? Or can this be prioritized (please)?

[smarshall-bmr]

@dlebauer We tend to get help from Headwall in bursts so I'm try to take advantage of their help while we have it.

@rjstrand Who do I need to talk to about the 50% grey target again?

Our current plan is to calibrate the VNIR on a 50% target, does this have to be a spectralon target or do we have other options on material? The calibration run will now include the SWIR as well, I haven't heard if there is a consensus on a target for that calibration. Maybe we should have a call to talk about that like we have with the VNIR? Of course I can just gather every target I can get my hands on and scan them all on the assumption that one will work for our purposes, or at least inform what type of target we should be using.

[dlebauer]

@smarshall-bmr makes sense. I think that the 'every target you can get your hands on' approach makes sense. From the summary of a previous call https://github.com/terraref/computing-pipeline/issues/208#issuecomment-280513034 it sounded like both a 4% and a 50% target were available and would be sufficient, but that having a target in between these values would also be useful.

[smarshall-bmr]

@dlebauer It occurred to me that the 50% target may not be spectralon, hopefully @rjstrand is able to chime in here.

[czender]

The brand of, or material used for, the target is immaterial as far as I can tell. If a non-Spectralon target is used then we should (and will) reword some of the documentation and scripts to use a more generic term (e.g., "target") than Spectralon, so as to prevent confusion.

[rjstrand]

Hi, Sorry for my absence from this conversation...

@dlebauer @smarshall-bmr As I remember, Nadia has money budgeted to buy panels specifically for this purpose. I think she is only waiting for someone to tell her how many, what size, and what reflectance values... within the budget constraints and then she can order them.

ALARC has panels and tarps. As I remember, the panels are 50 (Spectralon) and 20% (Not sure). Tarps are 68, 48, 8, and 4%.

@dlebauer - Stuart is out of town for a while. I'll contact you once I get to Maricopa next week.

[NewcombMaria]

The ALARC set of big tarps have been deployed near the scanner field on some of the UAV flight days around solar noon. The tarps are too large to place within the scanner field. While Stuart is out of town I've offered to help with the calibration scan, with Bob's help to get the correct targets in the correct positions etc.

[dlebauer]

@remotesensinglab will 20% and 40% suffice? Could you advise @nshakoor on which additional standards (if any) we will need?

@rjstrand thanks for your help! I understand we also these 95% reflectance and and color targets: SphereOptics-Diffuse-Reflectance-Standards-Targets-Materials-Zenith-Polymer.pdf spectralon_color_standards_data_sheet.pdf

[NewcombMaria]

@remotesensinglab and @dlebauer we are tentatively planning for April 15 for the 'clear sky diurnal hyperspectral spectralon images for calibration' scan. Weather permitting. @smarshall-bmr wrote a scan for sun-up to sun-down passes over targets before he left. We've borrowed spectralon targets from Kelly Thorp (ALARC). @remotesensinglab please advise on protocol. If Solmaz is able to bring the spectrometer with her, should the targets be validated using the spectometer at a couple time points (perhaps 10am and 2pm)? @rjstrand has offered to help and has been working with the SWIR and VNIR. We want to ensure that we follow the proper protocol since we may don't have many available sun-up to sun-down windows. Thanks.

[remotesensinglab]

@NewcombMaria this is great, the planning on April 15. I have given a list of targets to purchase to Nadia months ago. Having multiple targets are good (99%, 75%, 50%, 40%, and 2%), but 75% and/or 50% targets would be great if we have just one or two are available.

ALARC Tarps 68, 48, 8, and 4% may work but good to collect spectroscopy reflectance of these tarps as well just to confirm their reflectance. Yes, a 10am, 12PM, and 2Pm measurement of these tarps using the spectroscopy Solmaz has would be great.

Thanks

[dlebauer]

Everyone: very excited that there is a date for this, thanks for making it happen!

@remotesensinglab and @czender I'd like to confirm that we need to / plan to collect images of the spectralon targets at 15 minutes. I believe I saw a protocol written up somewhere but now am unable to locate said protocol.

[rjstrand]

@remotesensinglab - This is essentially the same set of targets that we used last fall, with the exception of replacing the 99% with yours. So, panels at 99, 75, 20%, Tarps at 68, 48, 8, 4%. I believe that some of these same tarps are documented in the attached paper published by my former USDA colleagues Moran, Clarke, and Qi.
2001_mar_273-286.pdf

[dlebauer]

@remotesensinglab The importance of a target near the max for vegetation (50-60%) has been repeatedly discussed. The paper that @rjstrand linked to shows that tarps are not substitutes for spectralon targets. Can you please confirm that the available targets are sufficient?

[czender]

Calibration targets should have reflectances in the range of plant reflectances. Targets that are too reflective (like the 99% Spectralon) provide information that is never used (unless you want to detect and produce reflectances for shiny metallic or specular surfaces that are also in the FOV). Pick calibration reflectances that bound the desired reflectances (so we get accurate brights and darks) and also have at least one calibration target near the mean- or median-expected reflectance. If we had only one target it would be optimal to have it there so that most of the pixels are well calibrated. With the luxury of multiple targets, choose them insofar as possible to be evenly spaced in reflectance between the maximum and minimum expected/desired reflectances (e.g., 10, 20, 30, 40, 50, 60%). The HS retrieval algorithm will do weighted interpolations between the nearest neighboring calibration values.

[czender]

Regarding temporal frequency of sampling, 15 minutes sounds good to me. It gives no worse than 360/(244) = 3.75 degree zenith angle resolution. It might be overkill if the zenith angle dependence is weak. But we've never measured it before so we don't know what to expect. There is no written protocol to my knowledge other than this, #208, and #289. And, to be clear, what I try to stress is that `max(plant_reflectance)exposure_time*max(counts_per_time) is ~2^16`. Otherwise we waste bits and reduce measurement precision.

[NewcombMaria]

Thanks @remotesensinglab @czender and @dlebauer for helpful input. We discussed our plan this morning and still have a couple questions: 1) Is it better to adjust the height of the sensor box so that the panels are in sun for much of the day? During the winter season and still at this time, the sun angle allows for sunshine in the field of view. However during the summer sorghum season with the change in sun angle, the field of view is under the shade of the sensor box. We can alter sun/shade exposure by altering the height of the sensor box, but the optimal height for best focus at this date puts the field of view mostly in the sunshine.

2) What camera exposure setting is best for the diurnal scan? We can use the same exposure settings used most recently (April 7-8) when Stuart ran a hyperspectral scan, which was an exposure for sunshine. What's the best exposure setting?

We will have to go with the targets that we have at this time point, which for the middle ranges include spectralon panels at 20% and 75%, and also tarps at 68, 48, 8 and 4%. We'll put out all the targets that we have, and if new ones are obtained in the future the diurnal scan can be repeated. We'll be able to get 15 min temporal frequency.

[czender]

Sun/shade is undoubtedly an important issue, perhaps as or more important as cloud/clear. I listed Sun/shade after cloud/clear in the #281 completion criteria for mainly subjective reasons. Ideally we will obtain one set of calibration images for direct sun and another with targets shaded by the sensor box, in both clear and overcast skies (i.e., 4 calibration images for each zenith angle, for each target, although 3 might suffice since the sun/shade distinction is much smaller on overcast days). We intend to use a sun/shade pixel mask (developed by @remotesensinglab?) to determine on which pixels to use the sun and which the shaded calibration. Is it possible on Monday to get a clean set of direct sunlight exposures (all target in sunlight) and another "clean set" of sensor-shaded (all target in shade) exposures?

Re: exposure setting. The calibration exposures are most useful when they are the same duration as the regular imagery. I though Monday's calibration exposures would be made at a range of durations that covers all the actual durations of the Hyperspectral exposures, e.g., 10ms, 15ms, 20ms, 25ms, 30ms. When we process a regular exposure we want to use the calibration exposure for the same amount of time and not have to interpolate/extrapolate to different times.

[NewcombMaria]

Thanks @czender. Our plan is do run the full-day scan tomorrow (Saturday) because it may be the only day before planting on the 19th and sprinkler irrigation after planting when we have clear skies and no equipment or sprinklers through the field. If we run the scan from sun-up to sun-down, we need to select either sun or shade exposure. Or do you mean run a sun-scan in the morning and a shade-scan in the afternoon? I know we have an opportunity Saturday the 15th, and there will be another opportunity for a second full-day scan at some point, but I don't know when.

Regarding exposure, typically the exposure is set for the entire script at one setting. I'm not sure what is meant by the range of durations of the hyperspectral exposures. Perhaps @rjstrand understands.

[czender]

@NewcombMaria If you can do sun in the morning and shade in the afternoon, please do. Then if nothing unforeseen happens we'll have both. Otherwise (if you can't do a half-day of each) please do a full day of sun. For the exposure, we want to use the maximum exposure for sun that does not saturate any wavelength at the maximum plant reflectance of interest. My understanding is that this is ~25ms for a maximum plant reflectance of ~60%. @solmazhajmohammadi and @remotesensinglab may know better.

[rjstrand]

@czender 1) In "max(plant_reflectance)exposure_timemax(counts_per_time) is ~2^16" can you please define counts_per_time

2) Ideally we don't want to move the targets between passes. I see two basic means of attaining predominant shade or sun on a target, either raising the camera box, or shifting the gantry on the x-axis to change the location of the target in the field of view of the cameras. Are there tradeoffs? it is possible that we will have to resort to raising the box

This is a lot to figure out/code/test in about 15 hours with no sun most of that time.

Maria just texted that at typical operating height, we are pretty well shaded at this point of the day. (almost 2 hours after solar noon).

i just picked up Solmaz and we will discuss on the way to the farm

[czender]

counts_per_time depends on the sensitivity of the camera and the incident sunlight. It can only be inferred from measurements. Pick the longest exposure that does not saturate the camera for the brightest sunlight and most reflective wavelength. This should be the same as the exposure time at which the cameras normally operate. If typical operation from now until October means that the FOV will be shaded, and you can't do both sun and shade in one day, then please perform calibration exposures for shaded targets. This is the opposite of what I suggested earlier when I thought there would be more sun. Guiding principle is to calibrate in conditions as close to normal operating as possible, hence do shade. That way most of each image will have a calibration in similar lighting conditions. If any part of FOV is sunny then exposure time must be set short (i.e., for sun) or else sunny portions of images will saturate and be useless. To switch between sun and shade it would be much better IMHO to shift gantry on x axis than to raise gantry.

[remotesensinglab]

Re: exposure time. 25 ms is good, no more than 30 ms per our analysis of data collected in Fall 2016 with 99% (?) target. If there is a room to collect longer exposure for experiment with the new calibration targets (20% and 75%), it is worth collecting at longer exposure times as well. So, we will have a better understanding of the matter, or confirm with the new calibration targets.

Re: Spectralon target. I thought we had more options for targets if the suggested targets were purchased per Bob's comment. It seems we have 20% and 75%. I would suggest using both if possible or 75%. @czender, can one of them be picked and used for calibration if one is more ideal for the sunlight conditions of tomorrow?

I do not know much about the tarps, but scanning them with the hand-held spectroradiometer Solmaz has would help to determine its reflectance, and correct use of the tarps during the calibration.

I agree that 15 min interval would be great, if can be done for the spectralon targets. This may not be necessary for scanning the tarps with hand-held spectroradiometer since that will be used to determine reflectances of the tarps.

@NewcombMaria, would you please clarify the shadow in summer and winter. Did you mean that:

• we have more shadow (due to the sensor box) in summer than in winter, OR
  • images in shadow is more than 50% of the images per full scan in sun during summer?

My point is that if images in shadow is still less than images in sun for one full scan of the field or for a day (even shadow in summer is more compared to winter), it may be good to collect it in sun if we have to stick to one. Plus, shadow may be compensated during level 2 processing.

Thanks

[NewcombMaria]

Thanks for responses. During the summer, there is more shadow due to the sensor box and we are able to get most or all of the field of view in shadow if we restrict day-time scans to approx 10am to 2:30/3pm. We talked as a group and have a plan for bracketing exposures and will do the best possible tomorrow. Thanks to @rjstrand and @solmazhajmohammadi and Ricardo for working on the scan script and making tomorrow's scan possible. We can repeat the scan again when we get other windows in the future.

[NewcombMaria]

The scan is going well and files are transferring to the cache server. During the mid-day hours the targets are in shade. See photo taken at 11:33am.
Solmaz wrote the script with a range of exposure settings. If it's possible to get a quick review of the data next week, we can modify the script and I'll watch for another clear-sky day without sprinkler irrigation to repeat the scan.

[dlebauer]

Just got the transfer stats and it looks like for SWIR we typically expect 1 hdr file per dataset but we're getting 5 per dataset. Is this an error?

On Sat, Apr 15, 2017 at 6:29 PM MariaNewcomb notifications@github.com wrote:

The scan is going well and files are transferring to the cache server. During the mid-day hours the targets are in shade. See photo taken at 11:33am. [image: diurnalhyperspec_april15_1133am] https://cloud.githubusercontent.com/assets/22008637/25067428/300e05fa-21f8-11e7-8536-7cecd53f2144.jpg Solmaz wrote the script with a range of exposure settings. If it's possible to get a quick review of the data next week, we can modify the script and I'll watch for another clear-sky day without sprinkler irrigation to repeat the scan.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/terraref/computing-pipeline/issues/281#issuecomment-294323768, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX55-BmOYjR90bKMk0oAU9qoKyP85fks5rwVLBgaJpZM4MXbGM .

[czender]

@NewcombMaria @solmazhajmohammadi What do we know about how level the targets were when imaged? Won't make much difference for shaded targets, but could be important for clear sky sunny targets. If/when this is repeated please measure any tilt with an inclinometer.

[dlebauer]

We can extract the angle from the RGB stereo camera. @pless can you process the target images from April 14 and provide a map of angles?

[NewcombMaria]

Thanks for input. The scan was April 15, started around 6:10am and stopped just before 7:00pm. The fixed metadata still says 'Wheat season', but the scan is named something like 'diurnal hyperspectral calibration'. It will be great to define what needs to be altered if there needs to be a repeat of the diurnal scan. Thanks.

[solmazhajmohammadi]

@dlebauer It seems that the sensor setting file was set to measure dark current, white reference and calibrate the data on the sensor machine and write back data cube in addition to raw data cube. For the data with 30ms and 35ms, you will see 14 files instead of 6. Following files can be deleted: XXXXdarkOffset XXXXdarkOffset.hdr XXXXdarkReference XXXXdarkReference.hdr XXXXdata XXXXdata.hdr XXXXwhiteReference XXXXwhiteReference.hdr

@czender we tried to level them, the wooden board has a little bow on it. I think the variation was less than 2-3 cm(from the center to the sides)

[smarshall-bmr]

First of all, big thanks to @solmazhajmohammadi @rjstrand and @NewcombMaria for doing this while I was away!

@czender I just wanted to clarify our operating times for being in the shadow of the sensor box. Because the hyperspectral cameras are located on the far west side of the sensor box they image in the shade for roughly 3.5 hours until a bit after solar noon, after that the imaging area is exposed to the sun. There's roughly a 1 hour period between when the VNIR gets out of the sensor box shadow and the SWIR does the same. This means that we can run shaded and/or exposed hyperspectral runs on the same day provided that runs fit in the 3.5 hour time frame.

[czender]

@smarshall-bmr Thanks, I'm starting to "get the picture" of the sun/shade exposure. I think if/when we re-calibrate we can do it more quickly by identifying and eliminating sub-optimal exposure times.

[solmazhajmohammadi]

@czender I was looking at the calibration data and I noticed that the position of the spectralon targets is not same all the times. This is caused by free running mode. The pause between the scans created few extra lines, so the targets will not show up exactly at the same pixel positions in all the hypercubes. The maximum shift that I have noticed was 100 lines, you will be safe if you are choosing few pixels in the center of the targets.

[czender]

Thanks @solmazhajmohammadi This shift will complicate the reduction procedure by requiring more person-time verifying the target locations. Nevertheless, it's good to know in advance that this will be required.

[dlebauer]

Can this be automated? From my naiive perspective, relative to other problems, it seems straight forward to clarify large homogeneous groups of pixels as one of n targets or other. On Tue, May 9, 2017 at 2:55 PM Charlie Zender notifications@github.com wrote:

Thanks @solmazhajmohammadi https://github.com/solmazhajmohammadi This shift will complicate the reduction procedure by requiring more person-time verifying the target locations. Nevertheless, it's good to know in advance that this will be required.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/terraref/computing-pipeline/issues/281#issuecomment-300282818, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX56wQMsqGMdGVW5SrZ6c2JFJQGU3Fks5r4MSfgaJpZM4MXbGM .

[czender]

@solmazhajmohammadi or @NewcombMaria or @smarshall-bmr Here is a representative unshaded image: /projects/arpae/terraref/sites/ua-mac/raw_data/VNIR/2017-04-15/2017-04-15__16-00-00-124/2e4f6f9d-2193-4f3c-8fcd-45010237a9cd_image.jpg Please post the name/model/factory-calibrated reflectance of each panel from top-to-bottom, left-to-right. We will manually draw bounding boxes. @dlebauer or @remotesensinglab anyone else please supply an automated technique if you have one.

[NewcombMaria]

@czender for the 20% and 75% panels and also the 48% small tarp, I've sent an email to the USDA researchers who lent us these targets. I'll post the info for these when we learn more. Stuart and Solmaz might have info on the remaining targets.

[czender]

@NewcombMaria @smarshall-bmr @solmazhajmohammadi We need a legend with the target name/reflectance written on top of each target in the picture. It's not obvious to me which target is which :) And of course to be accurate we need the full spectral factory calibration of each target. Currently we only have that for the nominal 95% reflectance Spectralon. Please upload all factory calibration data.

[solmazhajmohammadi]

[ghost]

@solmazhajmohammadi please add a link to the factory calibration information for the borrowed panels (20% and 75%).

[dlebauer]

Updates from slack

• on April 15 Solmaz captured a time-series of spectometer readings for all the targets using the spectrometer that she brought with her. Either Solmaz or Bob @rjstrand or both have the spectrometer data.
• The 75% and 20% are borrowed from Kelly Thorpe @kthorp at ALARC ... Kelly has measured them with his (very good) spectrometer. I'll try and get those readings for you.
• The grey tarps like the 48% are neutral grey for calibrating color cameras so they obviously won't have available hyperspectral data other then what Solmaz took.

[NewcombMaria]

Suzette and Kelly Thorp (USDA ALARC) provided the following information about the 20% and 75% panels. Both are from LabSphere. There's a calibration certificate for the 75% panel and an ID tag for the 20% panel. Both are attached here, along with the spectralon target specs from the website. spectralon_targets (from website).pdf

75_Panel Certificate.pdf

[dlebauer]

I've added data for the 75% target to the 'Target Calibration Datasets' folder on google drive. I noticed that the color target spectral reflectance response curves are also available in there, but those are only calibrated to 830nm

I also opened terraref/reference-data#132 to address compiling the calibration data and making it available.

[czender]

This is a good start. The 75% target data looks usable.

[dlebauer]

FYI I parsed the calibration table to CSV https://drive.google.com/open?id=0By_PDCcY5g2JWlU4R1VzMHFhejg

It is in the target calibration folder on Google drive alongside the measurements Solmaz made

Label here doesn't have calibration files for the 20% target. They suggested they might not be valid and could be recalibrated. On Thu, May 25, 2017 at 10:41 AM Charlie Zender notifications@github.com wrote:

This is a good start. The 75% target data looks usable.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/terraref/computing-pipeline/issues/281#issuecomment-304043275, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX56z4Ty3K0ZLKXxYmvfsbluTGKKfoks5r9aEegaJpZM4MXbGM .

[solmazhajmohammadi]

@czender I have not heard from you regarding the repetitive pattern in the target positions in the calibration files. I am tagging @FlyingWithJerome also, if he is still working on this. Please see the plotted output from the multiple scans below. The file size is changing, however, if you set the origin to the 20% target, the targets seems to be staying at the same position:

My first guess is that the file size and pattern should be same in the datasets with the same exposure time(or at least close enough that if you choose a ROI in the center of targets, we should be safe). Please try this, let me know how it goes.

[czender]

@hmb1 and @FlyingWithJerome can you please find the largest possible hyperslab for each target that remains on the target for the entire day (20170415)? It looks like it may not be possible for the 75% target which is relatively small, but might be possible for the 20%, 48%, and 95% targets. And keep @solmazhajmohammadi in the loop.

[hmb1]

@FlyingWithJerome, @czender is there any way I can view nc/image files on roger without downloading to my local machine as some of the files are huge ?

[czender]

I don't know of methods than ncview and panoply. Both should be able to display remotely from roger if your X DISPLAY environment is suitable. Maybe @ZongyangLi or @yanliu-chn know a better way.

[ghost]

@czender - Do you think this can be completed by then end of July? I want to make sure that it goes into the V1 release.