NerdWalletOSS / kinesis-python

Low level, multiprocessing based AWS Kinesis producer & consumer library
Other
118 stars 50 forks source link

Cannot be used from within Celery #17

Open danielcohenlive opened 6 years ago

danielcohenlive commented 6 years ago

See the stacktrace below

11:40:18 celery_general.1 | [2018-10-18 15:40:18,210: ERROR/ForkPoolWorker-2] Task xxx.apps.marketo.tasks.update_marketingdata_lead.update_marketingdata_lead[9f1605d0-bfe2-4549-8cf2-91abcdc8fd7b] raised unexpected: AssertionError('daemonic processes are not allowed to have children',)
11:40:18 celery_general.1 | Traceback (most recent call last):
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
11:40:18 celery_general.1 |     R = retval = fun(*args, **kwargs)
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
11:40:18 celery_general.1 |     return self.run(*args, **kwargs)
11:40:18 celery_general.1 |   File "xxxx/apps/marketo/tasks/update_marketingdata_lead.py", line 81, in update_marketingdata_lead
11:40:18 celery_general.1 |     _enqueue_lead_data(email, to_update, user_id)
11:40:18 celery_general.1 |   File "xxxx/apps/marketo/tasks/update_marketingdata_lead.py", line 93, in _enqueue_lead_data
11:40:18 celery_general.1 |     get_lead_producer().put(json.dumps(lead_data))
11:40:18 celery_general.1 |   File "xxxx/apps/xxxx_kinesis/__init__.py", line 39, in get_lead_producer
11:40:18 celery_general.1 |     buffer_time=5 * 60,
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/kinesis/producer.py", line 142, in __init__
11:40:18 celery_general.1 |     max_size=max_size, boto3_session=boto3_session)
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/kinesis/producer.py", line 78, in __init__
11:40:18 celery_general.1 |     self.start()
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/offspring/process.py", line 55, in start
11:40:18 celery_general.1 |     self.process.start()
11:40:18 celery_general.1 |   File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 124, in start
11:40:18 celery_general.1 |     'daemonic processes are not allowed to have children'
11:40:18 celery_general.1 | AssertionError: daemonic processes are not allowed to have children

My understanding is that using billiard instead of multiprocess will solve this issue