EPIC: Subject Matter Expert Function Support - Githubissues

cedardevs / onestop

OneStop is a data discovery system being built by CIRES researchers on a grant from the NOAA National Centers for Environmental Information. We welcome contributions from the community!

GNU General Public License v2.0

43 stars 21 forks source link

EPIC: Subject Matter Expert Function Support #1132

Open mcquinne opened 4 years ago

mcquinne commented 4 years ago

Summary

As a	subject matter expert
I want to be able to	trigger custom functions based on granule metadata
So that I can	extract rich metadata, populate custom DBs, etc.

Description

At long last! Time to support SME functions!

Use Cases

GIS: Populate a spatial DB with granule-level metadata to support map services
PAD: Support arbitrary, numerical fields for filtering, either in OneStop itself or a downstream DB
OER: Tag incoming files such that they can be easily found in the search API by a thin portal
CI: Trigger custom metadata extraction functions based on ingest stream and/or collection

Assumptions

We have a mechanism for grouping sets of granules together (#1124)
We are operating in an environment where serverless functions are available (e.g. AWS Lambda)

Tasks

[ ] Design a granule grouping mechanism with these use cases in mind (see #1124)
[ ] Determine how functions will be executed
[ ] Design data interface going to and from functions
[ ] Provide support for testing, e.g. test harness, dry-run capability, etc.

References

Point of contact	Jesse Varner (GIS), Charles Anderson (PAD)
WCSD Bucket Reference	https://ncei-wcsd-archive.s3-us-west-2.amazonaws.com/
PAD Bucket Reference	https://console.cloud.google.com/storage/browser/noaa-passive-bioacoustic

dneufeldcu commented 4 years ago

A new project level GCMD keyword has been established for PAD: SanctSound > NOAA-Navy Sanctuary Soundscapes Monitoring Project

Perhaps this can be used as the initial grouping.