asas-sn / skypatrol

The ASAS-SN Sky Patrol python client
GNU General Public License v3.0
15 stars 4 forks source link

Downloading light curves crashes if run inside multiprocessing environment #14

Open karpov-sv opened 1 year ago

karpov-sv commented 1 year ago

Hi,

The functions that download the lightcurves (i.e. cone_search) are crashing if run from inside multiprocessing environment, i.e. process pool. Originally encountered it while trying to use the library from inside Celery, but it also manifests in simple examples like that:

from pyasassn.client import SkyPatrolClient
import multiprocessing

def fn(x):
    client = SkyPatrolClient()
    lcs = client.cone_search(ra_deg=296.02191444, dec_deg=23.44665476, radius=10/3600, catalog='master_list', download=True, threads=1)

with multiprocessing.Pool(processes=1) as pool:
    pool.map(fn, [1])

Error message is AssertionError: daemonic processes are not allowed to have children, coming from inside _get_curves(), and the reason is what there another multiprocessing.Pool is being created, even if threads=1 is specified. And that is not seemingly supported by multiprocessing module.

It would be extremely nice if a separate code path, not involving multiprocessing, was used there when threads=1.

Thanks!