Daemonic processes are not allowed to have children

[pkikawa]

daemonic processes are not allowed to have children errror when uploading an .mp4 file. operation also left a uncommitted lock file in the collection. tested on latest release of ddrlocal-rc on Entiat. error gets thrown for all media types. tiff mp3 mp4 pdf jpg

see link: https://github.com/celery/celery/issues/4525

daemonicissue.zip

[gjost]

For some reason this is running DDR/vocab.get_vocabs which uses multiprocessing. This really should be run at startup but evidently it's not running until it happens in one of these tasks.

Traceback (most recent call last):
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/app/trace.py", line 472, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/opt/ddr-local/ddrlocal/webui/tasks/files.py", line 157, in after_return
    entity = Entity.from_identifier(Identifier(path=entity_path))
  File "/opt/ddr-local/ddrlocal/webui/models.py", line 535, in from_identifier
    return from_json(Entity, identifier.path_abs('json'), identifier)
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/models/common.py", line 811, in from_json
    document.load_json(fileio.read_text(json_path))
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/models/entity.py", line 434, in load_json
    json_data = common.load_json(self, module, json_text)
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/models/common.py", line 711, in load_json
    list(f.values())[0]
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/modules.py", line 112, in function
    value = function(value)
  File "/opt/ddr-local/ddr-defs/repo_models/entity.py", line 1080, in jsonload_topics
    converters.text_to_bracketids(text, ['term','id'])
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/vocab.py", line 699, in TEMP_scrub_topicdata
    for term in get_vocabs(config.VOCABS_URL)['topics']['terms']
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/vocab.py", line 637, in get_vocabs
    cached = _get_vocabs_all_http(vocabs_url)
  File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/vocab.py", line 609, in _get_vocabs_all_http
    return_dict = multiprocessing.Manager().dict()
  File "/usr/lib/python3.7/multiprocessing/context.py", line 56, in Manager
    m.start()
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 543, in start
    self._process.start()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 110, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children

[gjost]

I've tried this in my dev environment, and in an install from tgz in a fresh Debian 10 Buster VM and haven't been able to duplicate this. I'm using Celery 4.4.2 which is the version range mentioned in the Celery issue mentioned above.

Can you try something for me? Add this code to a new file in /opt/ddr-local/ddrlocal/webui/apps.py:

import os
from django.apps import AppConfig
from django.conf import settings
from DDR import vocab

class DDRLocalConfig(AppConfig):
    name = 'webui'
    verbose_name = "Densho Digital Repository Editor"
    def ready(self):
        if os.environ.get('RUN_MAIN'):
            startup()

def startup():
    # startup code goes here
    vocab.get_vocabs(settings.VOCABS_URL)

Add this line to the top of /opt/ddr-local/ddrlocal/webui/__init__.py

default_app_config = 'webui.apps.DDRLocalConfig'

Restart the app using sudo service supervisor restart and try the upload again.

[pkikawa]

Still getting the same error message when uploading binaries after making the modifications as suggested by jost in previous comment.

[gjost]

Could you try editing this line in /opt/ddr-local/Makefile,

celery -A ddrlocal worker -l INFO -f /var/log/ddr/worker.log

adding --pool=threads,

celery -A ddrlocal worker --pool=threads -l INFO -f /var/log/ddr/worker.log

and then running make runworker as the ddr user?

[gjost]

Should be fixed in ddr-local commit 5fa58b8 for package ddrlocal-rc_5.0.0-rc4.

[pkikawa]

fixed in 5.0.0rc4