search

kai3341 / celery-pool-asyncio

Celery pool to run coroutine tasks
https://pypi.org/project/celery-pool-asyncio/
Other
109 stars 12 forks source link

Running a group signature #34

Open satinfive opened 2 years ago

satinfive commented 2 years ago

Description

Im trying to use group to execute a list of subtasks inside of a celery task.

What I Did

I tried lots of variants in order to catch the result from the group. Not sure if I am doing something wrong or not, but anyway, group execution doesnt work. I saw on previous issues that this was fixed but its not working for me; not sure if it is because i am not doing it correctly or if its not fixed.

tasks = [task.s(var1, var2) for var1 in vars1] group = group(tasks) group_result = addr_group().get()

and then getting: TypeError: 'coroutine' object is not subscriptable

I tried using await, using apply_async, using delay, using a combination of all of them (await and apply_async), even iterating over the AsyncResult objects generated by group and trying to fetch the result using .get(), none of them worked. Its not allowed to use await, and everytime i try to get a result from the GroupResult or a single AsyncResult I got the error I reported above.

satinfive commented 2 years ago

Ok, i resolved the issue myself. Actually, this package supports group perfectly, but the code is in master and this is not released yet to pypi so if you install it through pypi it will not contain these changes to make group works.

Solution for now: install the package from the master branch in the repo instead of relying on pypi.

satinfive commented 2 years ago

It doesnt work. It works fine to retrieve a GroupResult using await but it fails trying to get the results. It fails first because an async get_many is returned to the iter_native iterator from result_set module. If you use an async for to iterate over the async iterator, then it stalls exactly awaiting for the drain_events :/