search

scverse / squidpy

Spatial Single Cell Analysis in Python
https://squidpy.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
401 stars 72 forks source link

Turning off multiprocessing completely #758

Open andrew-weisman opened 9 months ago

andrew-weisman commented 9 months ago

Hello, I'm running sq.gr.nhood_enrichment(adata, cluster_key=label_name, numba_parallel=False, n_jobs=1, backend=None) but multiprocessing still seems to get at least initialized somehow, which I know because I'm calling Squidpy from a program that's already employing multiprocessing and I get the error: "AssertionError: daemonic processes are not allowed to have children".

How to I completely turn off multiprocessing in Squidpy so that I can call Squidpy in parallel and won't get the error above?

Thanks in advance!

andrew-weisman commented 1 month ago

Hi @giovp @michalk8, just wondering, any updates on this? Thanks!

giovp commented 1 month ago

hi @andrew-weisman sorry for late reply, yet it is called but unfortunately there is no easy way to do it as multiprocessing is still initialized, I think the best way would be to use the helper function directly, see https://github.com/scverse/squidpy/blob/48bf4dd32d2ade76edcdc31f277ebd2a189402ea/src/squidpy/gr/_nhood.py#L187 and basically aggregate the results manually? otherwise, have you tried changing different backends https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html ?

andrew-weisman commented 1 month ago

Thanks for your response @giovp.

Okay good to know that multiprocessing is indeed initialized in squidpy which is probably why I'm seeing this.

Thanks for the ideas to try using the helper function alone or changing the backend called by sq.gr.nhood_enrichment(). Will keep these in mind, thanks. I imagined I tried the latter but I don't remember.

Would want to say it might be beneficial to you to allow the disabling (non-initialization) of multiprocessing in squidpy altogether so that other programs can call it if they are already using multiprocessing to speed up their pipelines of which squidpy is a part, as in our case.

Thanks!