Use DASK to iteratively parallelise the data loading and analyse large runs

iluise / atmorep_analysis

MIT License

0 stars 1 forks source link

Use DASK to iteratively parallelise the data loading and analyse large runs #3

Open iluise opened 1 month ago

iluise commented 1 month ago

Modify the atmorep_read_data interface to run with DASK arrays that can be easily parallelised. \ Bottleneck at the moment: we run out of memory when reconstructing more than 45000 local samples. This means ~ 20 dates in global forecast.

iluise commented 1 month ago

other possible alternative: https://developer.nvidia.com/blog/rapids-cudf-accelerates-pandas-nearly-150x-with-zero-code-changes/