search

eWaterCycle / infra

Instructions for system administrators to deploy the eWaterCycle platform
Apache License 2.0
0 stars 0 forks source link

ESMValTool memory issue in parallel computation #160

Open BSchilperoort opened 5 months ago

BSchilperoort commented 5 months ago

On the eWaterCycle teaching machines when generating forcing using LumpedMakkinkForcing the kernel freezes when the number of years > 4 (the juypter cell keeps running without producing a result) . It works fine up to 4 years of forcing.

After changing the ESMValTool configuration to use a sequential runner, the result is produced fast without issues.

stacktraces

![Screenshot 2024-04-02 at 12 49 53](https://github.com/eWaterCycle/infra/assets/12114825/8a69cea7-eade-417e-bdda-2eac4c30ab8b) ![Screenshot 2024-04-02 at 12 49 42](https://github.com/eWaterCycle/infra/assets/12114825/d102de5a-a05b-4a06-9a18-2785f26b94c0)

We should figure out if this is an ESMValTool problem, or if it's related to our (dask) configuration.

BSchilperoort commented 5 months ago

Issue found by @RolfHut last week, I decided to report it here for now.