search

SciTools / iris-esmf-regrid

A collection of structured and unstructured ESMF regridding schemes for Iris.
https://iris-esmf-regrid.readthedocs.io/en/latest
BSD 3-Clause "New" or "Revised" License
19 stars 17 forks source link

Larger-than-memory datasets with iris-esmf-regrid and dask #310

Open dennissergeev opened 12 months ago

dennissergeev commented 12 months ago

📰 Custom Issue

I was wondering if you have any recommendations on what dask settings I should use if I want to regrid a larger-than-memory dataset using iris-esmf-regrid.

The dataset is from an LFRic C24 run, containing about ten 2D or 3D variables with 1000 time slices loaded from 100 files (i.e. 10 time slices per file). The data are chunked accordingly: 100 chunks for every variable. The total size amounts to about 16G on disk.

I can obviously process this in a file-by-file loop but I hope to load the whole dataset and apply regridding in one go. Currently, my script halts because the regridding step consumes all available RAM. I understand this might be asking for too much specialised help but any advice would be highly appreciated!

Machine specs ``` OS : Linux CPU(s) : 8 Machine : x86_64 Architecture : 64bit RAM : 31.2 GiB Environment : Jupyter File system : ext4 GPU Vendor : Intel GPU Renderer : Mesa Intel(R) UHD Graphics 620 (KBL GT2) GPU Version : 4.6 (Core Profile) Mesa 23.0.4-0ubuntu1~22.04.1 Python 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0] ```
github-actions[bot] commented 5 months ago

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 180 days. Remove stale label or comment, otherwise this issue will close automatically in 14 days time.

github-actions[bot] commented 5 months ago

@SciTools-incubator/esmf-regrid-devs This stale issue has been automatically closed due to no community activity

dennissergeev commented 5 months ago

This is still relevant, so I would like this issue to be reopened. Thanks.

trexfeathers commented 5 months ago

@stephenworsley @pp-mo @HGWright any thoughts?

stephenworsley commented 5 months ago

I would have expected that if it's possible to do regridding file by file, dask should be able to handle the same data as a single dask array (with each file being a chunk) so I'm not entirely sure what could be causing problems. I wonder if dask is trying to run too many tasks in parallel, does this still blow memory if you are using a single threaded dask scheduler? For what it's worth, I've recently fixed a bug with the old way we were invoking dask #338, though I suspect this may be a seperate issue, it may be worth checking if the latest (unreleased) version of iris-esmf-regrid works any better. Otherwise, I'd be interested to see a sample of code which blows memory compared to a sample of code for which doing regridding file by file does not blow memory.