Publish a the chirp workflow and its elements to the Dockstore Catalog

[mike-gangl]

This is the creation/publication of the "entire" workflow for Chirp rebinning including its elements: Required Workflow Elements in Dockstore:

• search CMR for Chirp Data (16 days)
• stage-in all 16 days of data from the DAACs
• process it locally,
• Stage-out (write )data to a venue data bucket
• Catalog the data in the shared Unity Data Catalog

We can have a few version of this workflow overall:

1. Simple workflow to take in the 16 days (#59) of data and run it in the Application Package (Deployed to SounderSIPS Algorithm Test environment)
2. CMR based workflow + application package + stage-out
3. CMR based workflow + application package + stage-out + catalog (Deployed to SounderSIPS Operators Environment)

DAAC delivery is not a part of this deliverable

[rtapella]

How is the output data specified? Is it a DAPA Collection? Do we need to specify which "venue data bucket" to output to?

[ngachung]

Similar to Rob's question, what is the format of the output data and what is the format of the output metadata? UDS needs to know this so we can determine how to catalog.

[mike-gangl]

@rtapella the output data files are defined through the process within the application package and then staged-out; the result should be a STAC document that is available to other taks (e.g. data catalog). Not sure that answers your question or not.

[mike-gangl]

@ngachung they are netcdf i believe, metadata will be TBD- what required metadata do you need?

[rtapella]

Oh what I meant was: when you submit a job, you can tell the output where to go. I believe the idea is to specify which Collection it goes into. It could also be a s3 path or whatever, if the submitter is able to choose a physical location (s3, nfs, etc.) rather than a logical one (Collection).

Perhaps there is even a default Collection for each job, based on the jobID. And/or batch ID when we support that.