celery / billiard

Multiprocessing Pool Extensions
Other
419 stars 252 forks source link

ValueError: I/O operation on closed file when creating Pool #347

Closed Wats0ns closed 3 years ago

Wats0ns commented 3 years ago

Hello, we currently have a text analysis process running in docker. Part of it implies some text processing, which is run in a billiard Pool. However, it randomly raises the following exception:

Traceback (most recent call last):
  File "/project/tasks.py", line 37, in semantic_analysis
    worker.run(corpus_id)
  File "/project/text_worker.py", line 61, in run
    raise e
  File "/project/text_worker.py", line 35, in run
    corp.start()
  File "/project/sem/corpus.py", line 382, in start
    self.clean_content()
  File "/project/sem/corpus.py", line 178, in clean_content
    self.tokenized_content = self.text.parallel_tokenize(content, allowed_pos=self.allowed_pos_union)
  File "/project/sem/text.py", line 116, in parallel_tokenize
    pool = Pool(4)
  File "/usr/local/lib/python3.6/site-packages/billiard/context.py", line 165, in Pool
    context=self.get_context())
  File "/usr/local/lib/python3.6/site-packages/billiard/pool.py", line 1007, in __init__
    self._create_worker_process(i)
  File "/usr/local/lib/python3.6/site-packages/billiard/pool.py", line 1116, in _create_worker_process
    w.start()
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 124, in start
    self._popen = self._Popen(self)
  File "/usr/local/lib/python3.6/site-packages/billiard/context.py", line 333, in _Popen
    return Popen(process_obj)
  File "/usr/local/lib/python3.6/site-packages/billiard/popen_fork.py", line 22, in __init__
    sys.stderr.flush()
ValueError: I/O operation on closed file.

It looks like billiard is unable to flesh the stderr at the Popen object creation time. Any idea where this might come from ? My packages versions:

billiard==3.5.0.5
celery==4.1.1

Thanks for the help !

auvipy commented 3 years ago

please try celery=5.2.1 and report again. celery 4.x versions are EOL