mdbartos / pysheds

:earth_americas: Simple and fast watershed delineation in python.
GNU General Public License v3.0
702 stars 188 forks source link

Allow user to disable `parallel=True` with numba #234

Closed fmaussion closed 8 months ago

fmaussion commented 8 months ago

We, at https://github.com/OGGM/oggm, are using pysheds successfully to do useful stuff for glacier modelling - so thanks a lot for this great tool!

I'd like to be able to disable the use of parallelism in numba, which essentially involves replacing True to False in a few calls to numba.

The reason I need this is that we are calling pysheds on a large (>200k) number of small glacier domains and we are doing our own multiprocessing. We've found that numba then creates concurring processes and everything take forever (order of magnitudes more than when disabling multiprocessing in numba).

I don't know the codebase well enough to suggest the best way to do this cleanly within pysheds. Does pyshed have global variables, or run parameters? Or should we go by keywords in the whole API?

Thanks!

groutr commented 8 months ago

https://numba.readthedocs.io/en/stable/user/threading-layer.html#setting-the-number-of-threads

You can set the number of threads that numba uses with an environment variable. In your case, you would want to set NUMBA_NUM_THREADS=1

fmaussion commented 8 months ago

Thanks very much! That solves it very elegantly 👍