informatics-lab / precip_rediagnosis

Project to use ML to re-diagnose precipitation fields from ensemble model fields
0 stars 0 forks source link

improve scaling of data prep with dask #13

Closed stevehadd closed 1 year ago

stevehadd commented 2 years ago

creating a cluster might be helpful, for doing ML as well on azure ML https://cloudprovider.dask.org/en/latest/azure.html

arh89 commented 2 years ago

Agreed to put into backlog and re-evaluate whether necessary after @DPeterK's work with dagster

stevehadd commented 1 year ago

our thinking on how to do the data processing has probably movd on now, so would rather close this and pick up the data workflow stuff some other way