Open rswamina opened 6 days ago
Could you please share the complete recipe (except the diagnostic scripts if those need to stay confidential) so I can try to reproduce the issue?
careful extracting NSA :grin: :us_outlying_islands:
@bouweandela - Oddly enough this is working now. I don't know if this failed due to issues with JASMIN previously. I want to try a couple more things before closing this issue but will post back if I find a memory issue. Somehow I was sure I would need to use a dask configuration for this. If all works well, will close this issue shortly.
I am writing a recipe to extract all IPCC land regions (around 44 region extractions) and perform some basic preprocessing on them - regridding, temporal statistics. I need this applied for different seasons and multiple models. What would be the best way to do this and avoid memory issues? I find that including 10 models with 3 such preprocessors on JASMIN's sci machines makes the program hang. I am able to configure dask options but am wondering if I should split the region extractions across recipes or do fewer models or such as well ? An example preprocessor function with just one region would be:
@bouweandela - could you suggest how best to do this?