Review "cleaned" LemnaTec Field Scanalyzer metadata

[craig-willis]

As part of the terrautils implementation we've implemented a process to "clean" or standardize the metadata from the LemnaTec Field Scanalyzer. The "raw" metadata is now stored as a JSON file on ROGER and Clowder, but the Clowder metadata endpoint contains the cleaned metadata, which is now also used downstream by extractors.

Examples are available in the Data Release Trial space for the flirIR and stereoTop sensors.

FLIR:

• UI: https://terraref.ncsa.illinois.edu/clowder/datasets/59b062d94f0ca12ea0c32d04
• API: https://terraref.ncsa.illinois.edu/clowder/api/datasets/59b062d94f0ca12ea0c32d04/metadata.jsonld

StereoTop:

• UI: https://terraref.ncsa.illinois.edu/clowder/datasets/59b062d94f0ca12ea0c32d04
• API: https://terraref.ncsa.illinois.edu/clowder/api/datasets/59b062d94f0ca12ea0c32d04/metadata.jsonld

As noted in the PR, the primary goal was to clean up many of the inconsistencies in the raw metadata (i.e., Time v time v timestamp v Timestamp).

A few things to note:

• We discussed adding the sensor position in place of the gantry "position_m", but this has not been implemented
• Sensor fixed metadata is linked via URL
• I've added a dataset to capture the Gantry fixed metadata (https://terraref.ncsa.illinois.edu/clowder/api/datasets/5873eac64f0cad7d81349b0b/metadata.jsonld), which is linked in the metadata
• site_metadata was intended to get each of the plots associated with the image. As noted in 9/6 meeting, this is currently based on the centroid and should probably be removed or updated to make this clear.
• spatial_metadata was introduced to capture the bounding box and centroid. We've discussed changing to geoJSON and changing the current bounding box from format returned from terrautils.spatial _get_bounding_box_with_formula function
• spatial_metadata is intended to handle cases where there are multiple devices per sensor (e.g. left/right, east/west). For single-device sensors, the sensor name is currently used.

Completion criteria:

• [x] Metadata reviewed
• [x] Changes made for 2017 release
• [x] Issue(s) created for 2018 release
• [x] Changes identified and implemented in extractors as needed
• [ ] Extractors tested, tagged/versioned/deployed

[dlebauer]

Need to decide how to populate dataset owner, creator, terms of use, description

• Dataset owner: TERRA REF team
• Creator: extractor author(s)
• terms of use: link or text from documentation https://terraref.gitbooks.io/terraref-documentation/content/user/data_release_policy.html
• Description: perhaps some text concatenation of metadata?

[dlebauer]

Should mimetype be mac-binary?

[dlebauer]

For bounding box - we should calculate rectangle in x,y cordinates and then transform each corner to lat lon. The diversion from rectangularity can be assumed to be 0 in x,y and very small in lat-lon.

For now, we can assume that the cameras are all affixed to have FOV on the x,y rectangle.

[dlebauer]

Review of FLIR metadata https://terraref.ncsa.illinois.edu/clowder/datasets/59b062d94f0ca12ea0c32d04

• it isn't clear if the keys or the structure is defined,
  • e.g. what does "agent" refer to?
  • why is extractor_id under agent and extractor_version under content?
• add notes to site_metadata `"notes": "name of plot that contains image centroid"
• 2018: for experiment_metadata what we would be great is a url that links to the equivalent of select * from managements where treatment_id in (select id from treatments join traits join sites join experiments where experiments.name = ....)

[max-zilla]

@dlebauer

• Owner will be the Maricopa Site user we had as before (we can rename the display to whatever). I will see whether we can override creator to be a separate user, but that person must have a clowder account the extractor would have to hard-code (or pass in as an arg).
• Nadia had mentioned titles and I suggested "description" could have some fixed description for each data product. a concatenation of a small subset of metadata could work perhaps, maybe sensor, time, experiment, plot, ...?
• mimetype is auto-determined by Clowder
• @craig-willis and i plan to provide bounding box as GeoJSON with a simple method to convert between that and the current "array of numbers" format so extractors can consume with minimal code change.
• "agent" is a Clowder construct that says whether it was a user or an extractor that added the metadata. @robkooper intends to add a standard version suffix to the extractor_id so we dont need to put version in content, but that isn't available right now and we want to have the version SOMEWHERE easy to reach. eventually that'll go from 'content' to 'agent'.
• added a note to 'site_metadata' root level containing that text.
• agree on active bety link suggestion.

[dlebauer]

Is there a dictionary that defines metadata fields? For example, it is not clear what 'position_m' means - is it the position of a corner? The position of the center?

[dlebauer]

OK. now looking at the point cloud sample data https://terraref.ncsa.illinois.edu/clowder/datasets/59bac9c44f0c0b27bc3d3132 there is a positoin_m under 'gantry fixed metadata' and a speed. Then under sensor_variable_metadata there is a point_cloud_orign_m, a scan_speed_microMeter/s and a scan_distance_mm.

• these should all have consistent units. distance in m and speed in m/s. Also, it is not clear if point_cloud_origin is relative to the field [0,0,0] or otherwise.

[craig-willis]

@dlebauer Was there ever a dictionary from LemnaTec for the source fields? I expect the answer is no. Remember, most of these fields are simply an effort to standardize the information already provided by LemnaTec. If they didn't provide a dictionary, we haven't created one.

I can easily update the units on these fields (again, the units are the same as the source data provided by LemnaTec). We'll have to assess whether this impacts any downstream extractors.

The point_cloud_origin_m is something we've added to the metadata (similar to the spatial information) based on https://github.com/terraref/reference-data/issues/44. There seems to be some ongoing confusion over what this is. We should probably discuss in the call this week.

[dlebauer]

• this looks okay.
  • no more changes needed
  • added 2018 features to https://github.com/terraref/reference-data/wiki/Changes-for-2018#lemnatec-metadata
  • next step is for @max-zilla to deploy