It appears that data futures are not considered complete until the App that created them is complete. This is highly problematic for applications that might block for a very long time (hours) and produce intermediate output files along the way which are needed to initiate a stream of workflow tasks that should run concurrently with the still running app.
For example, lets say, app A runs for 6 hours. During those six hours, it produces an output file every 10 minutes. Each output file triggers a series of workflow tasks that should run concurrently with A. This allows pipelining of post-processing steps, for example, such that products from A's execution can be used, delivered, and evaluated while A is still running saving considerable time.
In the context of NWP, consider a forecast model writing forecast output during a simulation. Post-processing of the 1hr forecast should begin immediately upon it's output, rather than waiting for the entire (e.g. 120hr) forecast to complete.
It appears that data futures are not considered complete until the App that created them is complete. This is highly problematic for applications that might block for a very long time (hours) and produce intermediate output files along the way which are needed to initiate a stream of workflow tasks that should run concurrently with the still running app.
For example, lets say, app A runs for 6 hours. During those six hours, it produces an output file every 10 minutes. Each output file triggers a series of workflow tasks that should run concurrently with A. This allows pipelining of post-processing steps, for example, such that products from A's execution can be used, delivered, and evaluated while A is still running saving considerable time.
In the context of NWP, consider a forecast model writing forecast output during a simulation. Post-processing of the 1hr forecast should begin immediately upon it's output, rather than waiting for the entire (e.g. 120hr) forecast to complete.