Share / harmonize pipelines for greenhouse systems

[dlebauer]

Perhaps these should be separate issues, but I could imagine one developer from each interested team work on teaching each other how to use / deploy their tools. At the end a good summary of similarities and differences, and opportunities to share the effort.

Compare, contrast, exchange methods for extracting raw images from LT systems in order to work with images outside of LT software.

• Done independently by UNL, Danforth, Arkansas. Share lessons learned and how to generalize / extend to non-LT systems
  • UNL: We can extract all images prior to any LT processing and send directly to HPC center. This involved an additional server for the 3D greenhouse - server also serves as a backup. We can do this for the scanalyzer without purchasing an additional server
• Also done by Danforth / PlantCV: https://github.com/danforthcenter/pheno-data-service
• Define a general format for output that can be used by any automated system including a simple camera+tripod

Deploy PlantCV, pipeline at Arkansas, Purdue, and or UNL

• Optionally using Clowder for file storage and workflow automation
• Optionally inserting data into BETYdb or exporting at some standard format (determined in issue #1)

Exchange hyperspectral pipelines b/w TERRA REF, UNL, other users.

• how much generalization is necessary?
• TERRAREF has a specification for sensor metadata and data formats. Are these useful to others? If not, we can change them so please provide feedback.

Deploy Arkansas metadata capture front end to the Lemnatec database at Arkansas, UNL, how to

• Jack/Chris: is this interface a good starting point for #2 (extending Lemna control)
• Details: Chris Angel is developing an interface that may be used to conform to MIAPPE guidelines to capture experimental metadata for HTP experiments. Metadata is associated with Scanalyzer DB, from which raw images are extracted, along with the Lemnatec produced CV-based

[westerman]

@dlebauer You mentioned "...TERRAREF has a specification for sensor metadata and data formats. ..." Is there a link to these specs? Thanks.

[dlebauer]

@craig-willis can you please summarize/ link to your work?

[craig-willis]

This is a work in progress, but we've started centralizing the collection of fixed sensor information including specifications, documentation, and metadata in the TERRA-REF sensor data repository (Clowder). An early approach to fixed sensor metadata with some description is in github (https://github.com/terraref/sensor-metadata). The initial version is based on a format provided by LemnaTec for the Field Scanalzer, but we've been working to provide a general specification as more sites come online.

Please note that this is really focused on fixed (authority) metadata about devices and sensors, intended to capture specific properties/characteristics that might be useful for algorithm developers downstream.

[nfahlgren]

I would be interested in participating in this group at the hackathon. Reiterating what @dlebauer said above, I would love to have a setup something like the following at the Danforth Center:

1. Users of our phenotyping system(s) register their experimental design/metadata before the actual experiment using the Arkansas web portal (at Arkansas or local implementation), using the MIAPPE (or similar) standards.
2. When accessing data for analysis, we have a tool (extended from ours and/or the UNL tool) that can access the data in the LemnaTec database and the Arkansas portal, and locally stores the image and metadata in a standard way.
3. PlantCV, Image Harvest, etc. can read the local store of data and metadata for analysis.
4. Output results, traits, metadata, etc. are standardized as much as possible for integration with other analysis tools, BETYdb, back into the Arkansas portal, etc.

Other thoughts to add on to this:

• Experiments from the Arkansas portal could be "published" in a publicly discoverable way.
• Phenotyping systems could be registered with the Arkansas portal, using @craig-willis sensor metadata specifications.