pySTEPS / pysteps

Python framework for short-term ensemble prediction systems.
https://pysteps.github.io/
BSD 3-Clause "New" or "Revised" License
467 stars 169 forks source link

Optimisation in excprob function #424

Closed viktor40 closed 2 months ago

viktor40 commented 2 months ago

Currently pysteps.postprocesing.ensemblestats.excprob copies the input array only to replace all values afterwards by either a 0or a 1. It is much more efficient to just initialize an array of the same shape using np.zeros(). By doing this, the code can also be simplified by removing the part of the code setting values to 0.

Below, is example code showing the difference in performance between the two methods. This is for a 700x700 grid with 12 timesteps and 24 ensemble members. This shows a significant difference. The specific part of the code is sped up by four orders of magnitude, going from taking 153 ms to 20.5 µs.

import numpy as np
s = (24, 12, 700, 700)
test_arr = np.random.randint(1., 5., size=s)

%timeit test_arr.copy()
>>> 153 ms ± 4.87 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit np.zeros(s)
>>> 20.5 µs ± 656 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.87%. Comparing base (953f799) to head (30faaf2). Report is 10 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #424 +/- ## ========================================== + Coverage 83.52% 83.87% +0.35% ========================================== Files 159 160 +1 Lines 12575 12779 +204 ========================================== + Hits 10503 10719 +216 + Misses 2072 2060 -12 ``` | [Flag](https://app.codecov.io/gh/pySTEPS/pysteps/pull/424/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pySTEPS) | Coverage Δ | | |---|---|---| | [unit_tests](https://app.codecov.io/gh/pySTEPS/pysteps/pull/424/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pySTEPS) | `83.87% <100.00%> (+0.35%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pySTEPS#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.