celery / billiard

Multiprocessing Pool Extensions
Other
417 stars 252 forks source link

billiard.pool map function getting hang #258

Open vikramTungsten opened 5 years ago

vikramTungsten commented 5 years ago

I need to run my logic parallelly using all available CPUs, for that I am using billiard package for pooling, PFB my technology stack -

-Python 3
-Django==2.0.5
-Django rest from work
-celery==4.2.0

My problem is, when I am hitting my rest-API and triggering my celery task it executes till pooling line but after that, its getting hang, I am not getting any error message but It is not moving further.

PFB my project structure:

-cdm
  |cmdapp
  |  |
  |  |helper
  |  |  |aaa.py
  |  |tasks
  |  |  |bbb.py
  |  |api
  |     |ccc.py
  |cdm
  | |settings.py
  |manage.py

PFB my code example:

#cmdapp.helper.aaa.py
from billiard.pool import Pool
class A():
    def squre(self,number):
        return number*number
    def calculate(self,number_list):
        pool = Pool()
        result = pool.map(self.squre, number_list)
        return result

#cmdapp.tasks.bbb.py
from celery import task
from cmdapp.helper.aaa import A
@task
def calculation_task(number_list):
    a_obj=A()
    result=a_obj.calculate(number_list)

#cmdapp.api.ccc.py
from rest_framework.views import APIView
from rest_framework import status, permissions
from rest_framework.response import Response
from cmdapp.tasks.bbb import calculation_task

class C(APIView):
    def get(self, request):
        range_list=range(1000)
        calculation_task.delay(range_list)
        return Response({
            "result": "success",
        }, status=status.HTTP_200_OK)

I went through some articles where they have mentioned unpickable function cannot be called by pool.map function. Here I am not getting my function is pickable or unpickable and why It is getting a hang. Please help in that.

auvipy commented 5 years ago

could you try celery 4.3 and show the traceback error message?