MASS Extract Workflow - Githubissues

informatics-lab / precip_rediagnosis

Project to use ML to re-diagnose precipitation fields from ensemble model fields

0 stars 0 forks source link

MASS Extract Workflow #69

Closed DPeterK closed 1 year ago

DPeterK commented 2 years ago

A dagster workflow that encapsulates the logic needed to extract an archive file from MASS into a temporary directory, extract the archive, then unzip extracted files of interest into a final storage directory.

Not done yet:

apply this workflow to other MASS archives (such as model data). I'll make the changes necessary to get this workflow to support both archives in a follow-up PR.
skip execution if the archive file from MASS has already been retrieved.

DPeterK commented 1 year ago

The radar data getter workflow is now fully working!

DPeterK commented 1 year ago

Thanks @stevehadd for your review! As you note, a lot of your comments relate to docs and understanding of dagster. Now there's quite a lot potentially to document there (!) but certainly a quick overview of the key ideas - and a description of how to run the workflow (!) I will add to a README. Where comments directly relate to dagster comments I'll also try and add a dagster docs link that explains the concept.