NOAA-GSL / ExascaleWorkflowSandbox

Other
2 stars 2 forks source link

How to create Data Future without listing the file in "outputs" parameter? #16

Open christopherwharrop-noaa opened 1 year ago

christopherwharrop-noaa commented 1 year ago

In many cases, NOAA workflows have data dependencies on data that arrives from an external source throughout the day. The data are produced by some other entity and thus cannot be listed as "outputs" of an App. Is there any other way to create a data future that can be used to feed downstream tasks?

One possible solution would be to create a "wait for the data" App that simply polls for the data and terminates when it is found. This App could then list the data in its outputs, and downstream workflow tasks could thereby use it.