search

cemac / SWIFT-Testbed3

ARCHIVED. Repository for Testbed 3 Work Flow
GNU General Public License v3.0
2 stars 0 forks source link

Streamline pre-processing of GFS data #46

Closed tdjames1 closed 2 years ago

tdjames1 commented 2 years ago

Pre-processing stage involves converting to netcdf and combining forecast data.

Prototyping on JASMIN suggests that converting files serially takes ~90 minutes. Need to find a way to process more quickly. I have tried sending jobs to LOTUS as a batch array but this results in delays due to time in queue.

tdjames1 commented 2 years ago

For the dry run I set up scripts to do task farming for both pre-processing and plotting which means that the overall processing time meets the requirements of the synoptic forecasters (making 00 UTC GFS data available by 05 UTC since that would give 8 Local time in Kenya).

Still need to tidy up, document and check in workflow scripts.

tdjames1 commented 2 years ago

Workflow scripts added in PRs #62 and #70.

Closing this but note that a further improvement to the workflow would be to use rose/cylc suite to manage tasks.