search

pySTEPS / pysteps

Python framework for short-term ensemble prediction systems.
https://pysteps.github.io/
BSD 3-Clause "New" or "Revised" License
467 stars 169 forks source link

Implement the STEPS model in the spectral domain #78

Closed pulkkins closed 4 years ago

pulkkins commented 5 years ago

The AR(2) models and the noise generation in STEPS can be implemented in the spectral domain. See

S. Pulkkinen, V. Chandrasekar and A.-M. Harri, Stochastic Spectral Method for Radar-Based Probabilistic Precipitation Nowcasting, Journal of Atmospheric and Oceanic Technology, doi: 10.1175/JTECH-D-18-0242.1.

I implemented the above method in the spectral branch. Here is the output of the old version of nowcasts.steps.forecast by using the FMI data (grid size 760x1226 pixels) with 24 ensemble members and 12 threads:

Starting nowcast computation. Computing nowcast for time step 1... 8.15 seconds. Computing nowcast for time step 2... 8.23 seconds. Computing nowcast for time step 3... 8.11 seconds. Computing nowcast for time step 4... 8.05 seconds. Computing nowcast for time step 5... 7.90 seconds. Computing nowcast for time step 6... 8.04 seconds. Computing nowcast for time step 7... 8.17 seconds. Computing nowcast for time step 8... 8.11 seconds. Computing nowcast for time step 9... 8.09 seconds. Computing nowcast for time step 10... 7.94 seconds. Computing nowcast for time step 11... 7.94 seconds. Computing nowcast for time step 12... 8.07 seconds.

The memory usage was around 26% (on a computer with 50 Gb memory). Using the spectral implementation, I got the following results:

Starting nowcast computation. Computing nowcast for time step 1... 4.48 seconds. Computing nowcast for time step 2... 4.66 seconds. Computing nowcast for time step 3... 4.65 seconds. Computing nowcast for time step 4... 4.54 seconds. Computing nowcast for time step 5... 4.50 seconds. Computing nowcast for time step 6... 4.48 seconds. Computing nowcast for time step 7... 4.51 seconds. Computing nowcast for time step 8... 4.43 seconds. Computing nowcast for time step 9... 4.46 seconds. Computing nowcast for time step 10... 4.48 seconds. Computing nowcast for time step 11... 4.58 seconds. Computing nowcast for time step 12... 4.52 seconds.

Memory usage was reduced to 16%.

There are still rough edges and not everything is working. The only noise generation method implemented in the spectral domain is the non-parametric one. Once all bugs have been fixed and the other noise generators have been implemented in the spectral domain, I would recommend using the new version as the default choice. The classical STEPS model should still be kept in pysteps for educational purposes (and also for the localized versions where the localization is done in the spatial domain).

dnerini commented 5 years ago

This looks great, a fantastic improvement in performance! I wonder whether the the forecast quality is somehow affected? If not, I agree that this should become default.

pulkkins commented 5 years ago

The forecast quality should not be affected, as the forecast model remains the same. It's just implemented in a different domain. The only difference I can see is that the spectral version uses only one mean value for the whole precipitation field across all scales, whereas the original version estimates the mean value separately for each cascade level. According to my experiments, the mean value for all cascade levels beyond the first one is always very close to zero, so the difference is not significant.

dnerini commented 4 years ago

I suppose #133 closes this issue?

Great piece of work @pulkkins !