search

mpvginde / pysteps

Python framework for short-term ensemble prediction systems.
https://pysteps.github.io/
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Parallellized ensemble run seems to lose track of ensemble number and order #20

Closed RubenImhoff closed 9 months ago

RubenImhoff commented 11 months ago

If I run an ensemble using multiple cores (so, dask will parallellize the ensemble members over the cores), it seems that the ensemble order is lost, resulting in weird transitions (due to noise from a different member that ends up in that member). See for instance (these are three 15-min instance in the forecast): 2023-08-17 08_37_39-P_fc in 202308160545_blended_nowcast_ensemble_newc 2023-08-17 08_38_00-P_fc in 202308160545_blended_nowcast_ensemble_newc

2023-08-17 08_38_07-P_fc in 202308160545_blended_nowcast_ensemble_newc

Or maybe even clearer: MicrosoftTeams-image

Has any of you come across this and/or know how to fix it?

If I run on 1 core, this problem does not occur, so it gives the impression that this has to do with parallellizing using dask.

RubenImhoff commented 11 months ago

Found out that this is most likely an overall pysteps bug in the fft_method "pyfftw". Recommended to use "numpy" as method now for multi-threading. I think we can continue this issue here: https://github.com/pySTEPS/pysteps/issues/337.

Anyway, during my search, I have made a few code changes as pushed here: https://github.com/mpvginde/pysteps/commit/7912e2f96bdfab8499ce00c76aa6b5722b61dbe5. @mpvginde, can I ask you to have a quick review of these changes (sorry, perhaps I should have gone for a PR).