search

NOAA-GFDL / fre-workflows

Code to generate, describe, validate, and configure scientific workflows within the FRE software framework
2 stars 4 forks source link

statics processing not correct #8

Open ilaflott opened 3 months ago

ilaflott commented 3 months ago

The requested static components in the README do not get processed correctly, example: Screen Shot 2024-08-07 at 12 59 11 PM

no jobs should be running at this point, other than stage-history jobs, as those are what make the data available for later steps. Therefore, from the screenshot, what we can see is that e.g. for years 0001 and 0002, before the files are done being staged (stage-history:submitted), we have jobs attempting to run on those files (00020101T0000Z/remap-pp-components-ts-P2Y_land_static:running and 00020101T0000Z/remap-pp-components-av-P2Y_land_static:failed). Additionally, given that static data is time-indep, no time-series no time-averages should be being created.

ceblanton commented 3 months ago

Thanks Ian, I agree this is not right. How does FRE know that the land_static component contains only statics? It doesn't, you're finding.

There was a clunky way to do this using the remap-pp-component's rose-app.conf, but not yet in the pp.yaml. In the rose-app.conf settings, there was an optional freq setting, that could be used to only remap a subset of the source history files (e.g. monthly but not daily). I used that (freq=P0Y) to handle the static processing of those cases.

ceblanton commented 3 months ago

One way we could solve this is using the diag_out.yaml "history manifest" file during the Install step.

If the history files mapped to the land_static component are all static, then tasks for timeseries and timeaverages wouldn't be created.