Recovering Summer 2017 FLIR data and preventing future errors

[dlebauer]

Description

During review of the FLIR data products in https://github.com/terraref/reference-data/issues/182, we identified problems with data starting in May 2017.

FLIR data from summer 2017 has errors and artifacts due to sensor contamination. It appears that at some point early in the season, water got inside the camera, evaporated, and left behind a salt residue on the lens that is affecting a significant portion of most, if not all of our FLIR images.

@pless @remotesensinglab and @ZongyangLi: once the physical issue with the camera is resolved, we will need to see if there is are any methods that can be implemented to recover the images. The FLIR data from the BAP is a key dataset.

Details

Images can be viewed in Clowder (with an account) using the search bar

• for all FLIR images https://terraref.ncsa.illinois.edu/clowder/search?query=flir or for a specific date * for FLIR images from a certain date (here 2017-07-30) https://terraref.ncsa.illinois.edu/clowder/search?query=flir+2017-07-30

The order of tasks and priorities is important.

1. First priority is to identify any images we need to take before cleaning the camera that will aid in recovering data.
   • if the data can be recovered, how can this be done? What steps do we need to take before cleaning the camera?
2. Second priority is to determine what data we can include in the Nov 2017 release
   • if pre-contamination what is the cutoff date
   • for post-contamination, is there a clear way to mark the data as being in error? I am reluctant to publish data with such a severe known issue.
     • are there any simple steps we can take to remove the largest sources of errors (e.g. the image edges?)
   • Recovering data that won't be in the 2017 release is lower priority

Next steps:

• [x] Determine cutoff date for thermal data to be released in Nov 2017
• [ ] talk with FLIR and / or other experts to determine if the images can be recalibrated/recovered.
• [ ] draft protocols for
  • [x] any pre-cleaning calibration images
  • [ ] cleaning the camera, recalibrating, and preventing this (both contamination and delayed identification of the problem) from happening again
• [x] Take any pre-cleaning calibration images
• [ ] Clean and seal the camera

[dlebauer]

From @NewcombMaria

The time-point of concern seems to be April 30, with a good image on April 29 and a problematic image on May 1.

First we checked mid-March during the wheat crop.

The first image below is March 17. Sorghum planting was April 20.

The night the sprinkler intersected with the 3D scan was the night of April 27. The next image below is April 28, which looks ok.

Next is April 29 which also looks ok, although there's some banding (shade from the gantry beam?). April 29 shows tiny seedlings that emerged on April 26. There are no images from April 30.

Then May 1 someone already converted to color scale png in Clowder, clearly a problem with the image which persists through May 13.

April 30 did not have rainfall (according to Azmet). We still don't know the cause, but we've narrowed the timepoint.

[smarshall-bmr]

I'm working further on the idea of taking one set of pics through the crud with a uniform background then diffing out the background once the lens is clean to create a sort of filter that we could subtract from the old images. Right now I'm leaning towards painting a metal plate matte black to reduce IR reflectivity and floating it in an ice bath so it stays the same temperature then taking a time series so that the crud can change temperature. The FLIR provides a few different temperatures in the metadata so hopefully we can find one that correlates with the intensity of the signal from crud.

[dlebauer]

@smarshall-bmr are you drafting the protocol somewhere that @remotesensinglab (and others) can review and contribute to it?

[smarshall-bmr]

@dlebauer I've been tinkering with the FLIR to learn more about any features that it has that would be useful to us, the most useful thing I have found is that the autofocus can be overridden and set for a specific distance. This is actually incredibly useful because it allows me to put a standard target much closer and thus be much smaller. It also makes the FLIR "focus through" the target making it appear much more uniform.

[smarshall-bmr]

The draft protocol is available here:

https://drive.google.com/open?id=1YCSGLPzTmtUOuh0DOWDwlelXU2sBbsdZzPRut0MbIV8

[smarshall-bmr]

@dlebauer Would you or someone you know of be able to pull the autofocus metadata from the period of time when the FLIR was compromised? I want to be able to match the lens geometry as closely as possible when I characterize the resolution drop we have been experiencing.

[dlebauer]

@smarshall-bmr do you want the autofocus data from every image for the entire season? Or for specific dates / locations?

If you are interested in individual files, you can use the GUI thus:

• go to https://terraref.ncsa.illinois.edu/clowder/
• type something like "flir 2017-07-30" in the search bar
  • equivalently: https://terraref.ncsa.illinois.edu/clowder/search?query=flir+2017-07-30
• Then select an image, e.g. https://terraref.ncsa.illinois.edu/clowder/datasets/59b04d224f0ca12ea0bdfa0c
• select metadata tab
• the autofocus can be found under 'sensor_variable_metadata'

You can similarly query the metadata for the dataset in json format here: https://terraref.ncsa.illinois.edu/clowder/api/datasets/59b04d224f0ca12ea0bdfa0c/metadata.jsonld

[smarshall-bmr]

@dlebauer I'd actually like to know if there's a "consensus" focal length that I should be doing my testing at to most accurately match the conditions of the FLIR during the season. To do this I'd like to look at all the data.

[dlebauer]

I've scanned all of the images for probably 50 dates (find ua-mac/raw_data/flirIrCamera -name *.json | xargs grep AutoFocusand

I haven't seen any values other than 1, so I think it is safe to say that this is the consensus value recorded in the metadata, though I think it would be prudent to confirm that the actual value is always fixed (as opposed to the alternative which is that autofocus automatically focuses but these changes are not recorded in the metadata).

[dlebauer]

Sorry - are you looking for the "Manual focal length" and / or the "focal distance"?

[smarshall-bmr]

This is an image taken with a uniform ice water bath at right around 0C as a background as well as a histogram of the temperatures across the image. With this low temperature background the measurement ranges from 18C to 28C. Unfortunately this means that there isn't an area of the FLIR image that isn't effected by the crud, but we are hoping that with some more tests we still might figure out a way to run a correction.

[dlebauer]

From a small sample of dates / times the focus distance seems to vary from 1-2 m; sometimes it is not provided. It would take a while to find all of the unique values or the range. I can do it but if you just need a consensus then 1.5m is a safe bet.

Perhaps it would be better to just work on one or two sample dates. Otherwise, it might be worth doing doing scans at 1, 1.2 1.4, 1.8, 2 m, depending on how difficult the scans are to do.

Has @remotesensinglab had a chance to review the protocols that you wrote up?

[dlebauer]

So the sensor is biased +20C and estimates a constant temperature with a range of 10C. The signal we are looking for is << 1C differences in the leaves, and I suspect the data primarily reflect the temperature of the lens.

It is difficult for me to imagine how we could get any biologically relevant information out of this dataset.

[JeffWhiteAZ]

Another test various people have suggested is simply to take a large stack of images from a single scan and calculate the mean temperature for each pixel position. If the stack avoids obvious sources of bias such as differences in illumination and amount of soil in view, the pixel means should be pretty uniform across the image. This approach could also be used to test for effects of air temperature, time of day or trends over days (e.g. changes in "crud" distribution or texture). Ideally, the bias would be invariant with time of day, temperature and date, and a single correction would be attempted. Looking at the variation and each pixel position might tell us if we have also lost sensitivity.

[anfrench]

im wondering if the calibration routine, or the coefficients being used are inapplicable for cold temperatures; getting 20C bias is way out of expectation for the FLIR. so a suggestion: can the data be acquired using the FLIR proprietary format, then converted using FLIR's ResearchIR package? this confirm or refute those highly inaccurate temperatures over the ice bath.

[remotesensinglab]

@smarshall-bmr would it be possible to take hundreds of (about 300 hundreds FLIR images) for calibration of this issue with and without salt on the lens cover?

The protocol you put together is good. A couple of points: 1) I think @dlebauer 's suggestion on collecting data from different distance is a good point. 2) Any chance for collecting the images for real plant/soil targets in addition to what you proposed? 3) Have the images been collected in radiometric JPG, 14-bit tiff, or JPG?

Thanks

[smarshall-bmr]

@dlebauer In the metadata the Autofocus has a value of 1 on 0 depending on if the autofocus is active. The manual focal length is poorly named, it should be the current focal length setting.

@anfrench makes a really good point here, I don't know if we have ever compared the output of the FLIR converter that I am using to data converted "natively" by FLIR software. We have measured the blackbody in the past to pretty high accuracy though.

@JeffWhiteAZ and @remotesensinglab are talking about the same thing, using image statistics to determine the signal from the crud from a "batch" of images. I think this will be the best way to extract relative temperature values.

@remotesensinglab to answer your other questions, I'll take measurements at different distances from the target. What could we gain by measuring plant/soil targets right now? The images have been collected in .bin format

[NewcombMaria]

Thanks Stuart for getting some test images yesterday! It's not easy to gather known-temp targets of some size and I'm glad to continue to help out as needed. I've copied the FLIR images taken yesterday from the Cache server since I don't see them at clowder yet. They are now in a Google directory folder 'FLIR test images 2017-10-11' at https://drive.google.com/drive/folders/0B1OR5-zeXagnY3ZTTFNZcGRIQm8

For each time directory the .bin and metadata files are included for the following tests: 1) an ice-water bath and 2) ambient-temp water with the FLIR camera fairly close to the targets so they covered the field of view. Also 3) the black/white checkerboard target at 2 meters. Suggestions for known-temp targets that can be scanned at multiple heights up to 2 m are welcome. Thanks to everyone for contributing to the discussion and to finding as best solution as possible.

[smarshall-bmr]

Here is an image of a water bath with a similar setup to the ice water bath, but this time with the water at an ambient temperature of 33.1C, as well as an image from the handheld FLIR from the same perspective. The departure from actual temperature as well as the range is much smaller in this case.

[Paheding]

Aiming to retrieve useful information from noisy (salt-affected) FLIR images, we have tried to denoise few image data captured on May 01 to May 16, the input noisy data with the corresponding out images after denoising are shown below. From these results, it can be observed that the plant is not visible from the FLIR data that were captured from May 01 and May 04. I assume this is due to the majority area of the lens is blocked by salt residue which may lead to the emitted energy from the plant cannot be captured by sensors.

May 01, 2017

May 04, 2017

May 16, 2017 ( the image is getting better, leaf is visible except there are still some noise and blurring artifacts)

[NewcombMaria]

@smarshall-bmr if convenient, and if you have access during your travel, will you share any new Flir images since 2017-10-11 and update either here or slack? Thanks.

[NewcombMaria]

@smarshall-bmr Last week Pedro, Bob and I talked about the FLIR status and decided it was best to move forward by collecting the required Flir images with the protective lens deposits in place, removing and cleaning the cover, then collecting duplicate comparative images with the lens cleaned. John, Bob, and I set out targets and collected these images last Thursday Nov 9. The .bin and metadata files are now both on the Cache server and have been transferred to NCSA.

As a brief summary:

• targets were a large-area ambient water target that John created (we've been referring to it as John's pool), leveled field soil, and a checkerboard (black and white)
• 3 total sets of images were captured: with deposits on lens cover (3 heights), with lens cover cleaned (3 heights), and without the lens cover (1 height)
• ground-truth temps were collected using thermocouples for the water, handheld Apogee IRT for field soil, and the spot camera for the checkerboard. I still need to write up these data first chance I get.

The FLIR camera and protective cover have been removed and Bob re-wrote the RMA for shipment. If it hasn't already been shipped off today, it will be soon. I think it's known from our previous discussions that I believe the data recovery effort is critical. I remain an optimist that our interdisciplinary team can come together and recover data with scientific value, even if it's relative indices of some sort to look for genetic differences in thermal regulation among BAP lines. Let me/us know if there are any questions.

[NewcombMaria]

Thanks @remotesensinglab (to both Patrick and Wasit) for working on thermal data recovery. @anfrench is also generously offering to help. Andy has processed some of the images (converted .bin to .png for viewing, and converted to temp data (units K) files but so far without atmospheric correction). @smarshall-bmr can also fill in information about this issue.

I've put the unprocessed and processed images in a Google folder within TERR-REF directory. The main folder is called 'FLIR test images 2017 data recovery' at directory https://drive.google.com/drive/folders/0B1OR5-zeXagnY3ZTTFNZcGRIQm8

Within that directory, there are 2 folders for the Nov 9 images (2017-11-09_unprocessed_CacheServer and 2017-11-09_processedno_atmo_AFrench). Each image file has it's own folder by timestamp with the .bin and with the metadata file. We started taking test images around 13:35, and continued through about 15:55. (Andy, you ended up with images taken before our target tests when we were testing autofocus - sorry about that.) We took images at 3 different heights (see metadata for z=2.25 which translates to approx 3.5 m from target to FLIR lens; z=1.5 m which translates to 2.75 m from target to FLIR lens; and z=0.75 which translates to approx 2.0 m from target to FLIR lens). For each set of images we stayed at the same height over all 3 target (water at ambient temperature, field soil with tire tracks, and a black/white checkerboard).

We have 'ground-truth' temperatures of the water (thermocouple data), soil (handheld IRT data), and checkerboard (Flir thermal spot camera). I need to organize these data and will get them accessible to everyone soon.

Let me know if you have any questions. Also let me know if you have any troubles accessing the Google folder. I have the images on a thumb drive and could find another way to get them to you if the google directory is a problem.

[NewcombMaria]

Another important piece of information is the time intervals for the different sets of images:

• first with the deposits still in place on the protective lens (images from approx time 13:36 to 14:37).

• Then there's a time gap when the lens was removed and cleaned. We took images with the protective lens cover completely removed at only one height (z=1.5 m) taken from time 14:58 to 15:08.

• Then the cleaned lens cover was put back in place for another set of images taken from time 15:20 to 15:55

[NewcombMaria]

Notes from Nov 9 FLIR image collection with and without deposits on the lens cover are now copied to the shared folder, along with the FLIR TG167 handheld spot camera images of the checkerboard target and a spreadsheet with the weather station data for afternoon of Nov 9.
https://drive.google.com/drive/folders/0B1OR5-zeXagnY3ZTTFNZcGRIQm8

[NewcombMaria]

Andy French shared an update on his progress working with the FLIR images captured Nov 9. Thanks Andy!

Update from Andy: "To independently check the atmospheric property terms in the matlab code I ran MODTRAN simulations ( I can provide associated files if needed) using the local weather data information from the scanalyzer along with a Tucson radiosonde profile, the latter was collected a long ways away from MAC, but it was the closest available atmospheric profile. I then filtered the MODTRAN output using a FLIR-supplied spectral response function (attached) to get the following values for a camera to target distance of 2m: Tau (atmospheric transmissivity: 0.9841; Lup (upwelling atmospheric radiance between the surface and the camera in mW/[m^2 sr micrometer]): 87.961; Ldn(downwelling atmospheric radiance in mW/[m^2 sr micrometer]): 3730.905; using emissivity of 0.98 reflected Ldn is 74.618

When I use the FLIR-supplied estimates I get: Tau: 0.995; Lup: 72.667; reflected Ldn: 288.393; Compared to MODTRAN, this is a high transmissivity, about the same upwelling spectral radiance; and a high reflected downwelling spectral radiance. The exact impact from using one method or the other will have to wait until we verify the FLIR camera units, but roughly the correction should lead to increasing the apparent FLIR temperature readings by ~1.2 Celsius. Still to check are the raw FLIR temperatures vs. the water temperatures."

Note: The function document and net response document referred to as attached are saved as .txt docs in the google folder at directory https://drive.google.com/drive/folders/0B1OR5-zeXagnY3ZTTFNZcGRIQm8

[Paheding]

@NewcombMaria Thanks for providing those data and associated information. Here is one question regarding data collection. For images with pool targets in all three sets of images, were they taken in the same scene with all pixels are aligned (i.e., pixels are overlapped at the same spot in all images sets) from each other?

[NewcombMaria]

Patrick, regarding the pixel overlap for the pool target images, we returned to the same X and Y coordinates for all 3 sets of images but it's unlikely that the positioning system is precise enough to exact alignment of the pixels. However it's probably fairly close to the same pixel positions. Maybe @smarshall-bmr knows more about the precision of the positioning.

[Paheding]

@NewcombMaria Thanks, I think you are right. I've checked few sets of images, it looks like they are pretty much aligned.

[Paheding]

We did some analysis on the data provided by the team. The results are summarized as follows:

1. We first computed a correlation between the lens with and without deposits for pool target. This helps us to determine if there any pattern can be found and learned by a machine learning technique for data recovery. For pool target, determination of coefficient (R^2) is found to be 0.87, the corresponding scatter plot is shown as follow:

2. Applied the trained model to a test image (2017-11-09__13-41-54-077), the output looks like this:

It can be seen that the difference between the input image (lens with deposits) and the recovered image is obvious. However, it cannot be ensured that the generated output is the expected or not since the reference image (i.e., ground truth) at that exact or very close time slot is not available. If there are two sets of images (one from the lens with deposits, another from cleaned lens) taken by a short time interval (e.g., within one minute), it could be helpful to evaluate the recovered image and build a robust model.

[NewcombMaria]

Thanks very much Patrick to you and your colleagues for working on this issue! Unfortunately there are no images within a minute comparing the target with and without deposits on the lens, although the pool of water was large enough mass that the temperature did not change very rapidly over the day. We also have 'ground-truth' temperatures from thermocouples. Have you converted to actual temperatures, or are these relative values in the image? Do you have a sense of how close the recovered data are to the water temp recorded by the thermocouples? It's exciting to see progress on recovery steps and analysis. Thank you.

[Paheding]

@NewcombMaria Thanks for your input. Those values are directly from raw images, I have not converted them to actual temperature yet. Is there any specific formula for temperature conversion? I can try to compare the results using the 'ground-truth' temperatures from thermocouples. Please share the 'ground-truth' data whenever it's ready.

As for the accuracy, it is really hard to determine since small changes of temperature on the targets could affect the prediction accuracy. I looked at the relative values in the images, the range is relatively small. However, the contaminated images from this summer seem to have a variety of ranges of values. Once I got the temperature converted and 'ground-truth' data, I can make a relative assessment of the recovered images.

[NewcombMaria]

@Paheding the 'ground-truth' temperatures for the pool and soil targets are in the document 'Notes_ReferenceTempData_2917_11_09_flir.docx' in the google folder 'FLIR test images 2017 data recovery' at https://drive.google.com/drive/folders/0B1OR5-zeXagnY3ZTTFNZcGRIQm8 I'll also send the document by email.

Regarding the conversion to temperatures, @anfrench has worked on the conversion (see Nov 24 comment in this issue). It would be good for you to connect with Andy about his progress with the conversions to raw FLIR temperatures.

[Paheding]

@NewcombMaria Thanks, I will look into those documents.

[Paheding]

@NewcombMaria I am just wondering about the temperature conversion. Shall I send you and Andy a sample output for performing conversion? And then, we can check if the recovered data are reasonable or not.

[NewcombMaria]

@Paheding it's good timing to connect with @anfrench about temperature conversion. Andy is guiding our in-house FLIR calibration procedure and he is currently working on the data processing and analysis. I'll connect you with Andy by email for now and we can update the github later with summary results.

[Paheding]

Based on our discussion, the reference data for temperature evaluation: https://github.com/terraref/reference-data/issues/197