search

RolnickLab / ClimateSetExtension

Data Processing Pipelines for ClimateSet - can be use to build your own climate model datasets for machine learning
0 stars 0 forks source link

Downloader refactoring #10

Open mjfortier opened 1 month ago

mjfortier commented 1 month ago

This is meant to be an ongoing PR that all of us may end up committing to. The primary objective is to refactor the downloader to accommodate a small test dataset.

After a lot of experimentation, it seems that the only way this is possible is to filter the wget script at runtime; pyESGF does not appear to have affordances for this.

mjfortier commented 1 month ago

So this works well for CMIP6, but we're running into an issue for Input4MIPS where the grid_labels array is empty, and the downloader seems ill-equipped to handle that. image