Implementing basic functionality for dummy run

[JimCircadian]

Make sure the implementations all work for the following use cases. We previously know that the downloading worked for daily records across complete hemispheres, so here we account for that, but while we're doing so identify the requirements for new monthly and new sub-hemisphere configurations.

We only use some demonstration variables to ensure that things are operating correctly.

HEMI=north
WORKERS=1
PERIOD_START="2021-11-1"
PERIOD_END="2022-1-31"

download_cmip -w $WORKERS -v --source MRI-ESM2-0 --member r1i1p1f1 $HEMI $PERIOD_START $PERIOD_END uas,vas,tas,zg,siconca ',,,500|250,'
download_cmip -w $WORKERS -v --source EC-Earth3 --member r1i1p1f1 $HEMI $PERIOD_START $PERIOD_END uas,vas,tas,zg,siconca ',,,500|250,'

download_cmip -w $WORKERS -v --source MRI-ESM2-0 --member r1i1p1f1 $HEMI $PERIOD_START $PERIOD_END thetao,so,uo,vo ',,,'
download_cmip -w $WORKERS -v --source EC-Earth3 --member r1i1p1f1 $HEMI $PERIOD_START $PERIOD_END thetao,so,uo,vo ',,,'

download_era5 -w $WORKERS -v $HEMI $PERIOD_START $PERIOD_END uas,vas,tas,zg  ',,,500|250'
download_oras5 -w $WORKERS -v $HEMI $PERIOD_START $PERIOD_END thetao,so,uo,vo ',,,'

download_hres -w $WORKERS -v $HEMI $PERIOD_START $PERIOD_END thetao,so,uo,vo,uas,vas,tas,zg,siconca ',,,,,,500|250,'
download_seas -w $WORKERS -v $HEMI $PERIOD_START $PERIOD_END thetao,so,uo,vo,uas,vas,tas,zg,siconca ',,,,,,500|250,'

# Purely Sea Ice, siconca
download_osisaf -w $WORKERS -v $HEMI $PERIOD_START $PERIOD_END 
download_amsr2 -w $WORKERS -v $HEMI $PERIOD_START $PERIOD_END 

Tasks:

• [x] All downloaders should require --vars / --levels (or sensible default that is obvious from CLI)
• [x] Refactor the structure of the base downloading, this needs to change
  • [x] Remove the assertion that SIC is the reference dataset for downloaded files
  • [x] Get the basic downloading from all sources working, without postprocessing
  • [x] Move regridding against reference data into a subclass that is not SIC dependent
  • [x] Move rotation of wind data to a mixin / subclass for use with appropriate downloaders
  • [x] Call these via the postprocessing interface
• [x] Sea variables downloaders don't tend not to use levels and previously CMIP didn't provide this. We need to make this explicit and consistent
• [x] Review the horrendous ESGF downloading as it never really worked nicely (perhaps we should use THREDDS?)
• [x] #10 ...
• [x] Raise issue(s) and identify requirements for monthly downloading
• [x] Raise issue for location based downloading #6
• [x] Assess whether the directory structure needs changing, for example to remove hemisphere. This might be shifted towards data generation within the downstream icenet library. We might need to look at condensing of datasets / ensuring compatibility with existing data when dataset IDs clash

[JimCircadian]

download_cmip is failing, usually happens with existing mode of download from ceda.ac.uk but can't guarantee it's still the same. Sorting out the implementation method

[26-01-24 20:13:08 :ERROR   ] - Error encountered: [Errno -70] NetCDF: DAP server error: 'https://esgf.ceda.ac.uk/thredds/dodsC/esg_cmip6/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r1i1p1f1/day/uas/gn/v20190603/uas_day_MRI-ESM2-0_ssp245_r1i1p1f1_gn_20150101-20641231.nc'

[JimCircadian]

Multi-temporal downloading has been completed across CMIP, CDS, OSISAF and AMSR. Regridding is now a concern of preprocess-toolbox, as well as preprocessing such as rotation of fields. Levels have been implemented as much as required for those downloaders. An initial assessment of the data directory structure is in place, but open to change - if that changes we probably need to ensure that conversions can happen, so dataset configs need some versioning / structuring as well.

This basic functionality is in place, but more to do under the subsequent issues.