omnilib / aiomultiprocess

Take a modern Python codebase to the next level of performance.
https://aiomultiprocess.omnilib.dev
MIT License
1.77k stars 101 forks source link

how multi processes can be used for each task? #90

Open doncat99 opened 3 years ago

doncat99 commented 3 years ago

Description

I test the following two methods, method_1 can show multiprocessing is involved for task dispatching. While in method_2, all tasks are involved in a single process, can anyone show me a way out for involving multiprocesses in method_2? Thx in advanced.

Details

async def get(url):
    async with request("GET", url) as response:
        result = await response.text("utf-8")
        logger.info(len(result))
        return result

async def method_1():
    urls = ["https://jreese.sh", "https://noswap.com", "https://omnilib.dev", "https://jreese.sh", "https://noswap.com", "https://omnilib.dev", "https://jreese.sh", "https://noswap.com", "https://omnilib.dev", "https://jreese.sh", "https://noswap.com", "https://omnilib.dev"]
    async with amp.Pool() as pool:
        async for result in pool.map(get, urls):
            logger.info(len(result))

async def method_2():
    pool_tasks = []
    calls_list = ["https://jreese.sh", "https://noswap.com", "https://omnilib.dev", "https://jreese.sh", "https://noswap.com", "https://omnilib.dev", "https://jreese.sh", "https://noswap.com", "https://omnilib.dev", "https://jreese.sh", "https://noswap.com", "https://omnilib.dev"]
    async with amp.Pool() as pool:
        for call in calls_list:
            pool_tasks.append(pool.apply(get, args=[call]))
        [await _ for _ in tqdm(asyncio.as_completed(pool_tasks), total=len(pool_tasks), ncols=90, desc="total", position=0, leave=True)]
doncat99 commented 3 years ago

My Fault, it is only because the default childconcurrency set to 16, and my data_len is about 12, cheers~