search

pySTEPS / pysteps

Python framework for short-term ensemble prediction systems.
https://pysteps.github.io/
BSD 3-Clause "New" or "Revised" License
441 stars 160 forks source link

No reproducibility of steps blended nowcast when using noise_stddev_adj='auto' #346

Closed mpvginde closed 5 months ago

mpvginde commented 6 months ago

When running 2 identical blended nowcasts with pysteps.blending.steps.forecast using
a fixed seed
and setting
noise_stddev_adj = 'auto',
the results (sometimes) differ.

I suspect this behavior is coming from the pysteps.noise.utils.compute_noise_stddev_adjs function.

This function takes the seed argument, but inside the function the seed is always set to None: https://github.com/pySTEPS/pysteps/blob/be8eea432239f77bdfb7edf51a9bfaa94d012752/pysteps/noise/utils.py#L102-L106

Leading to different normalization weights for every run and impacting the final results, preventing reproducibility.

Kind regards, @mpvginde

dnerini commented 6 months ago

Hi @mpvginde thanks for reporting this issue! Do you want to go ahead and submit a pr (looks like a simple fix)? Otherwise I'll try to have a look on it

mpvginde commented 6 months ago

Sure @dnerini. I'll have a look at it.

mpvginde commented 6 months ago

A small update: The issue might be more complex as I initially thought. When I fix passing the seed in the pysteps.noise.utils.compute_noise_stddev_adjs results become reproducable for the pysteps.blending.steps.forecast routine. However, for pysteps.nowcast.steps.forecast results still differ between identical runs. Only when I disable the dask parallel workers in the _update helper routine in nowcasts.steps the results become reproducable. I suspect this might be linked to issue #337, I will investigate further.

mpvginde commented 6 months ago

See https://github.com/pySTEPS/pysteps/issues/337#issuecomment-1935702000

Should be fixed with PR #347