Provide RSR curve data for hyper / multispectral imaging

[dlebauer]

Relative Spectral Response curves for all of the spectrometers

First column in each table should be the wavelength. Then, for each detector there should be a column (e.g. three cols for NDVI, four (?) for RGB). Presumably there is one for each of the following:

• [x] VNIR
• [x] SWIR
• [x] Skye NDVI pointing up
• [x] Skye NDVI pointing down
• [x] Skye PRI
• [x] Crop Circle proprietary, not provided
• [x] Quantum PAR sensor
• [x] RGB (not available per https://github.com/terraref/reference-data/issues/53#issuecomment-250465279)

Here are some examples from the documentation provided

SKYE NDVI

SKYE PRI

Ocean Optics

[TinoDornbusch]

@dlebauer Do you want to get the response curves as numeric values? Please check with @czender on the requirement of spectral sensitivity of Hyperspectral cameras. To my understanding the spectral sensitivity of the chip is elimated by referencing to the Spectralon White target.

[TinoDornbusch]

@dlebauer @czender Here is what I got from Labsphere regarding the Spectralon Target:

The target is not specifically calibrated. It is part of a production chain. Documents are in the master branch. Upload here was not supported.

[czender]

Please provide a specific hyperlink to the documentation you refer to. The calibration procedure requires numeric values for the absolute reflectance at each wavelength. Right now we assume 95% at all wavelengths. @TinoDornbusch we're looking for tabular values for VNIR and SWIR wavelengths.

[serbinsh]

If you received a reflectance calibration standard from Labsphere you should also have received a PDF and a small CD with the calibration coefficients to convert relative reflectance to absolute reflectance. And example PDF is attached here for one of my standards

[calibration coefficients moved to https://gist.github.com/dlebauer/a86c68f1c08f356205665d1a0e58f3b9]

[LTBen]

We did, the CD with the reflectance calibration is in Maricopa with the target, in a red envelope. I’m not sure if Bob has it already or if Stuart has access to it, either way we will upload the content of the CD soon.

Would it help for setting up the workflow to get the values from a different target in the meantime?

Best regards Ben

[TinoDornbusch]

Attached the ceriticate for Spectralon white target SphereOptics Panel.pdf

[TinoDornbusch]

Attached the spectral response of VNIR (G4-384) and SWIR (G4-383) we have got from headwall which corresponds to the certificate.

G4-383 & 384 Pixel to wavelength.xlsx

[TinoDornbusch]

Spectral characteristics of RGB camera not available according to AVT support:

Sehr geehrter Herr Dr. Dornbusch,

Das Quanteneffizienz-Diagram bekommen wir direkt von dem Hersteller, in diesem Fall OnSemi. Onsemi veröffentlicht keine numerische Werten für ihre Sensoren. Auf Seite 9 und 10 des entsprechenden Onsemi Manuals der Sensor, befindet sich (Link: http://www.onsemi.com/pub_link/Collateral/KAI-08050-D.PDF). Bitte darauf achten, dass ihre GT ein Sensor der 1. gen besitzt.

Mit freundlichen Grüßen,

i.A. Bernardo Luck // Applications Engineer

[czender]

@TinoDornbusch thank you for posting the spreadsheet and certificate. It appears on my MacOS application (Numbers) to show pixel vs. wavelength for both sensors. I thought there would also be a spreadsheet with the calibrated reflectance vs. wavelength at 1 nm resolution (based on file posted by @serbinsh). Instead, for reflectance, the only information I see is a poor quality photo of a PDF that shows a plot and has printed numeric values every 50 nm. We would like a file with all the data used to plot those curves. Am I missing something? please check the disk for the tabular spectral reflectances at 1 nm resolution. tagging @solmazhajmohammadi too.

[dlebauer]

@craig-willis please see https://github.com/terraref/computing-pipeline/commit/e6a540a6ac40bff4201d46c43bfd0f7a787ca81e for how the spectralon data are used in the computing pipeline, and where being able to query it on the fly would be of value.

@TinoDornbusch fig 6 provides the RSR curves for the RGB, no? Is AVI willing to provide the figure but not the raw data?

[solmazhajmohammadi]

@dlebauer Here is the response I got from Allied Vision:

Hello Solmaz, I regret it's not available in spreadsheet format. I guess you could create such a table "by eye and hand", by reading from the graph. Remember, it's just a sample measurement meant to be representative, so modest variations according to how pure your light source is etc. would all factor in anyway. Regards, Scott

And as I mentioned in today's talk Crop Circle is not providing this information. Proprietary ....

[solmazhajmohammadi]

Spectral Response Curve for PAR sensor : SQ-110 and SQ-500 Spectral response.xlsx

Models SQ-110 and 214 use the same detector. The SQ-214 curve is the one that cuts off at 655 nm.

[TinoDornbusch]

@czender This is all we got from headwall. Further we have no tabular data of the Spectralon target.

@dlebauer AVT gets these response curves from their chip supplier and they told me that they do not have the raw data.

[dlebauer]

This looks like it https://github.com/terraref/reference-data/blob/master/Zenith%20LiteT%20Diffuse%20Reflectance%20Target%20-%2095%25R.csv On Fri, Sep 30, 2016 at 1:46 AM TinoDornbusch notifications@github.com wrote:

@czender https://github.com/czender This is all we got from headwall. Further we have no tabular data of the Spectralon target.

@dlebauer https://github.com/dlebauer AVT gets these response curves from their chip supplier and they told me that they do not have the raw data.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/terraref/reference-data/issues/53#issuecomment-250671964, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX50oMyVbEwX_wQP157Wr3C8hLhrAcks5qvLBAgaJpZM4KC4rV .

[czender]

@dlebauer Yes! looks like @serbinsh received 50 nm more calibration, out to 2500 nm, but we'll take what we can get.

[solmazhajmohammadi]

RSR Curve for NDVI and PRI sensors

45951-45954.xlsx

[craig-willis]

@TinoDornbusch

I'm new to RSR curves and have been asked to convert the provided files into a common matrix format . Most of the provided spreadsheets report the response at a given wavelength, except for VNIR/SWIR. "G4-383 & 384 Pixel to wavelength.xlsx" has two columns Pixel and Wave. It's unclear to me how this relates to the RSR curve -- is this something you can help clarify for me?

[solmazhajmohammadi]

@craig-willis please disregard the VNIR/SWIR file. That's not a RSR curve

[dlebauer]

@solmazhajmohammadi what is it? Are the RSR curves for these sensors available?

[solmazhajmohammadi]

@dlebauer That's wavelength to pixel calibration data which is shown in the headwall camera's certificate. Headwall does not have the RSR curve for SWIR (not delivered from their suppliers), and for the VNIR, LemnaTec will sign a non-disclosure agreement to get the file.

[craig-willis]

@dlebauer I've put together an example Clowder dataset for the NDVI sensors with the RSR matrices embedded in the attached JSON.

In this case, there are two NDVI sensors (serial numbers 45951 top/up and 45952 down).

I've created a dataset in clowder-dev for the NDVI sensor with attached PDFs, etc. You can see the fixed metadata in the "Metadata" tab:

https://terraref.ncsa.illinois.edu/clowder-dev/datasets/5811034ce4b0d51359b959e8

The attached metadata is available via the metadata.jsonld endpoint. For example, loading the JSON data via R:

d <- fromJSON("https://terraref.ncsa.illinois.edu/clowder-dev/api/datasets/5811034ce4b0d51359b959e8/metadata.jsonld")
d$content$sensors[[1]]$serial_number
d$content$sensors[[1]]$rsr$data

Feedback welcome.

[dlebauer]

@craig-willis looks great! @ashiklom I think that the following would be identical to the rsr curves that you have:

rsr <- d$content$sensors[[1]]$rsr$data[[1]]
colnames(rsr) <- d$content$sensors[[1]]$rsr$dimnames[[1]]
head(rsr)
     wavelength CH1 CH4
[1,]        350   0   0
[2,]        351   0   0
[3,]        352   0   0
[4,]        353   0   0
[5,]        354   0   0
[6,]        355   0   0

[serbinsh]

I agree this organization looks great and is comparable to what we work with for other sensors, nice job! I really like how easy it is to pull the data from JSON in R

[craig-willis]

Hello @czender

One of my tasks has been to define a way to store reflectance data so that it's accessible programmatically and I could use your input on how I'm handling the SphereOptics Zenith target.

Similar to the approach taken for the sensor RSR data above, I've created a dataset to represent the SphereOptics Zenith Polymer target in Clowder that contains a product datasheet and the calibration certificate:

https://terraref.ncsa.illinois.edu/clowder/datasets/5818bb544f0c63f2a8d20963

I've added metadata to the "dataset" via the Clowder API in JSON-LD format: https://terraref.ncsa.illinois.edu/clowder/api/datasets/5818bb544f0c63f2a8d20963/metadata.jsonld

The basic JSON-LD structure is required by Clowder, but I've defined the "content" object contents. This includes basic metadata (manufacturer, serial number, etc) as well as a "reflectance" object with fields "wavelength" and "R" with values taken from

https://github.com/terraref/reference-data/blob/master/Zenith%20LiteT%20Diffuse%20Reflectance%20Target%20-%2095%25R.csv

I think this matches the matrix in your hyperspectral extractor.

At this point, I'm just looking for general feedback: Would the JSON API approach be useful to you? Does the structure make sense?

[czender]

Hi Craig,

Thanks for systematizing the Spectralon data. The JSON-LD structure is "metadata-poor", yet should be serviceable for some programs. If the goal is to make this site the one-true repository of the calibration data, then please add all the relevant metadata for the wavelength and reflectance fields that you find in the hyperspectral_calibration.nco file. Also, perhaps we should adopt the same names for the variables (i.e., factory_calibrated_reflectance). The JSON API is not yet useful to us to read-in the data, although it may become useful with more work on our end. This is the first I've seen about the "content" metadata. Perhaps we can make more use of that ourselves when we dump JSON.

Charlie

[craig-willis]

@czender Thanks for the feedback. I'll make the changes as suggested.

[ghost]

@craig-willis - can you please update this issue? Can a new issue be made?

[craig-willis]

@rachelshekar As far as I'm concerned, yes (I created #76 for actions I'm aware of). There are some unchecked boxes at the top and I'm not sure whether we're still waiting for data for any of the devices.

[ghost]

@dlebauer - are we waiting for data from VNIR, SWIR, and crop circle?

[dlebauer]

• Crop Circle RSR is not provided by the manufacturer.
• VNIR and SWIR RSR curves are in https://github.com/terraref/reference-data/issues/53#issuecomment-250443179. @craig-willis can you please confirm that this is captured in our sensor metadata. and if so, close the issue? (if not, add it?)

[craig-willis]

@dlebauer

Per Solmaz's follow-up comments, those are not RSR curves for the VNIR and SWIR. I missed her second comment that there is nor RSR curve for SWIR and that Lemnatec will not sign the NDA for VNIR.

So, no the file attached to that comment are not captured in the metadata. It wasn't clear to me whether the pixel-to-wavelength data was useful. It sounds like we will not have RSR for Crop-Circle, VNIR, or SWIR. If this is acceptable, and I don't need to do anything with the pixel-to-wavelength data, I'll close this.

[dlebauer]

@craig-willis I think it is acceptable not to have RSR curves for VNIR and SWIR. Presumably this is corrected in the calibration. If there is a demand / use case we can revisit this. We can likely generate RSR curves for these instruments with acceptable precision using the a radiometer and Spectralon target, or some other method. I am closing this issue.

However, I see @remotesensinglab hasn't been on this thread ... so I am pinging him and @Mamatemenrs FYI.