Try running the GEAR pipeline with dask/spark directly

NERC-CEH / dri_gridded_data

GNU General Public License v3.0

0 stars 0 forks source link

Open mattjbr123 opened 1 month ago

mattjbr123 commented 1 month ago

On JASMIN to start with

mattjbr123 commented 1 month ago

The Direct Runner seems to work fine on JASMIN LOTUS

mattjbr123 commented 1 month ago

@iwalmsley has a script for converting and rechunking multi-TB datasets to zarr with dask on JASMIN LOTUS, probably worth a look https://gitlab.ceh.ac.uk/zarr-data-access/zarr-conversion/-/blob/main/wrf/dask_slurm_1960.py?ref_type=heads