search

mdbartos / pysheds

:earth_americas: Simple and fast watershed delineation in python.
GNU General Public License v3.0
702 stars 188 forks source link

Numba JIT causes up to 15s import times #222

Open fluffbucket opened 1 year ago

fluffbucket commented 1 year ago

Compiling sGrid and whatever else uses Numba takes up to 15seconds every time causing long bootup times, consider using AOT instead with Numba.

Measured with cProfile around importing pysheds grid and raster

mdbartos commented 1 year ago

Hi @fluffbucket, all of the functions use numba's caching option to store the compiled code: https://github.com/mdbartos/pysheds/blob/32fb2efba94c5da4ee8e2556d484a218d352226b/pysheds/_sgrid.py#L11

It should just compile once the first time you use it and then not compile again unless modified.

fluffbucket commented 1 year ago

Indeed that's what it does, however even loading in the already cached methods takes that long which is why I reported it. Try measuring it.

On Wed, 21 Jun 2023 at 18:01, Matt Bartos @.***> wrote:

Hi @fluffbucket https://github.com/fluffbucket, all of the functions use numba's caching option to store the compiled code:

https://github.com/mdbartos/pysheds/blob/32fb2efba94c5da4ee8e2556d484a218d352226b/pysheds/_sgrid.py#L11

It should just compile once the first time you use it and then not compile again unless modified.

— Reply to this email directly, view it on GitHub https://github.com/mdbartos/pysheds/issues/222#issuecomment-1601111471, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATH2XB7WO6N4GC4TVLELOKDXMMLFNANCNFSM6AAAAAAZOIZ7HY . You are receiving this because you were mentioned.Message ID: @.***>

wiesnerfriedman commented 11 months ago

Hi @fluffbucket I'm wondering if you're using Windows? I'm having issues using from pysheds.grid import Grid and it seems to crash in _sgrid.py around def _d8_catchment_iter_numba. I'm thinking it's a Windows thing maybe?

mdbartos commented 11 months ago

I've noticed numba sometimes has problems on Windows with the functions that use numba implementations of linked lists and dictionaries. You may want to try using algorithm='recursive' in your call to grid.catchment and see if it's more stable.

I haven't been able to reproduce the 15+ second loading times though. I suspect that you may have a caching issue that is causing everything to be re-compiled each time you import.