search

informatics-lab / precip_rediagnosis

Project to use ML to re-diagnose precipitation fields from ensemble model fields
0 stars 0 forks source link

Create cylc suite to process extended dataset #73

Closed stevehadd closed 1 year ago

stevehadd commented 1 year ago

In the next phase of data prep, we will want to process data for a whole month or season at a time. The current scripts would require a lot of memory for this, so we should aim to do it in batches (I think 10 day batches should be OK). We could then have a rose suite to cycle through the dates. See the forsest upload suite for a template.

stevehadd commented 1 year ago

similar to other suggestions, I think I'd take a different approach to this now, so closing and can create new sssues in future about ideas for orchestrating and scaling the data prep workflow.