As an operator, I want to catalog and search on the provenance of data products

mike-gangl commented 1 year ago

There are a number of reasons why this is done:

a bug was discovered in a PGE and we want to reprocess all data that used it
an issue with upstream data (E.g. an ancillary product) and we want to reprocess data with that used it with a newer / corrected version
As an open science user, i want to try out a workflow with known inputs and outputs to determine reproducability

rtapella commented 1 year ago

not sure if this is product features or process

mike-gangl commented 10 months ago

Marked as Risk:

While discussing the custom metadata portion in U-DS, the distributed nature of "provenance" was discussed in the Unity environment. The idea of entity, activity, and agents come from multiple locations. Should the data product catalog include its own provenance? How can we link a given data catalog entry to activities that use it? this gets pretty complicated. We also need to consider the ephemeral nature of missions, and should this information be recorded for longer periods of time or should the archive be relied on for this.

mike-gangl commented 10 months ago

this came up regarding "labels", and maybe that the workflow label is not to be added to the product metadata, but a way of connecting the two is actually required. So instead of direclty linking a label to a product, we can search on a label, see all the jobs AND data output with it, and make the linkage external to both the processes and the data.

rtapella commented 10 months ago

Is there a LBL file or something like that where we could store metadata in a common way that travels with the files?

— Rob Tapella UX Designer 818-393-8030

From: mike-gangl @.> Sent: Wednesday, August 23, 2023 8:56:47 AM To: unity-sds/unity-project-management @.> Cc: Tapella, Rob (US 398H) @.>; Comment @.> Subject: [EXTERNAL] Re: [unity-sds/unity-project-management] As an operator, I want to catalog and search on the provenance of data products (Issue #47)

this came up regarding "labels", and maybe that the workflow label is not to be added to the product metadata, but a way of connecting the two is actually required. So instead of direclty linking a label to a product, we can search on a label, see all the jobs AND data output with it, and make the linkage external to both the processes and the data.

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https://github.com/unity-sds/unity-project-management/issues/47*issuecomment-1690219450__;Iw!!PvBDto6Hs4WbVuu7!O4sjaO9R3KXEDYp3qYc5aYSTa_KAu6tT_5eQz-mUtFWnavujySR97yECueAm0HTLGe_-sC3qw0Npfdwm1R-MXl9o9GrHRwcLMQs$, or unsubscribehttps://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAI2IYMELHSSEO6CESUFSP3XWYR37ANCNFSM6AAAAAAVZRAQKA__;!!PvBDto6Hs4WbVuu7!O4sjaO9R3KXEDYp3qYc5aYSTa_KAu6tT_5eQz-mUtFWnavujySR97yECueAm0HTLGe_-sC3qw0Npfdwm1R-MXl9o9GrHz_tdVOE$. You are receiving this because you commented.Message ID: @.***>

unity-sds / unity-project-management

As an operator, I want to catalog and search on the provenance of data products #47