Review existing standards, conventions, and ontologies. Which should we use, adopt, support, learn from?

[dlebauer]

Our goal is to create data products that are easy to access and use.

There are a few classes of data:

1. Crop physiological traits (#18)
2. Agronomic meta-data (#18)
3. Sensor Output (#14)
4. Derived metrics from image analysis (spectral indices like NDVI, geometric statistics like height, convex hull)
5. Genomics
6. (others? is this the correct list?)

For each class of data:

• what are the key existing standards / conventions?
• what is on the list or in the wild that we should not adopt
• what are general principles and where is minimum meta-data information content defined?
• what are the widely accepted 'unique identifiers' that we clearly should adopt ?
  • (e.g. UTC for time, lat lon for location, Binomial classification (Linnean Latin names) for species)?
• what are the use cases (queries that we should support)?

Each data format should have brief description, focusing on

• features relevant to TERRA
• existing tools and applications
• ease and value of integration / adoption / adherance
• general recommendation
• TODO
  • [ ] Update list of standards in documentation
  • [x] discuss in comments (below)
  • [x] summarize information and contribute or add to comment below

    Notes

This is a proposal open for comments and contributions. We plan to update these specifications annually, starting with v0 in Nov. 2016

[ghost]

@gsrohde - have you started on this yet?

[dlebauer]

@craig-willis please discuss with @gsrohde

[craig-willis]

Per discussion with @dlebauer, the remaining work on this task is to ensure that the data standards documentation is up-to-date based on discussions in the subissues.

[craig-willis]

@dlebauer is there a list of known/expected data products similar to the NEON catalog? If not, would it be worth me trying to put something like this together? I think it would be helpful to ensure full coverage of existing standards for this task. It might also be useful to map from the Field Scanner Data to the current locations in the Globus Endpoint.

[dlebauer]

@craig-willis

is there a list of known/expected data products similar to the NEON catalog?

not beyond what is already in the documentation, and not summarized in a table you linked, but this is a good idea.

map from the Field Scanner Data to the current locations in the Globus Endpoint.

That is a good idea, although I think the data products will either be in as-yet non-existent directories (not under raw_data/ua_mac/moving_sensors), and at this point I am not sure of the directory structure that Clowder users, and I don't think it is organized by sensor. So we should discuss 1) if it makes sense to provide access via Globus (beyond just to the terraref team) and 2) if we should ask Clowder to organize the files (or perhaps make a human readable directory of soft-links)

[craig-willis]

@dlebauer perhaps I could just maintain this as internal documentation for now. Is there a place we put internal docs, e.g., Wiki? I'm using the NCSA Confluence for my personal notes now.

Another question: I'm trying to map different processes (e.g., extractors) to the different data levels/products. Max walked me through 3 extractors: Geospatial, PlantCV, and Hyperspectral. Is there a list of current/planned extractors for the initial data product release?

[dlebauer]

@craig-willis You can use the documentation repository (github.com/terraref/documentation) or google drive for drafting this.

Is there a list of current/planned extractors for the initial data product release?

Not that I can point to right now - many exist as open issues in the computing pipeline repository but I can start writing one up.

[craig-willis]

@dlebauer I've been writing up notes from my analysis of this and #18 and #31 in a Google Doc

https://docs.google.com/document/d/13gXD_OVLffm0hqahDZ3tUvru8IV1fRfM6DiuOcfjr3s/edit?usp=sharing

and accompanying spreadsheet:

https://docs.google.com/spreadsheets/d/1qu3LYomGIiC4Lmzler-bH8jvyfR-GA6lDl5Go0NhO2c/edit?usp=sharing

They are still rough, but I wonder if any of it might be useful for the upcoming meetings.

[craig-willis]

@rachelshekar It's not clear to me whether I can call this "done" given the earlier work on https://github.com/terraref/documentation/blob/master/existing_data_standards.md

[craig-willis]

Closing this issue. Updating this documentation will likely be ongoing but can be incorporated into future tickets as needed.