Closed pkikawa closed 4 years ago
For some reason this is running DDR/vocab.get_vocabs
which uses multiprocessing
. This really should be run at startup but evidently it's not running until it happens in one of these tasks.
Traceback (most recent call last):
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/celery/app/trace.py", line 472, in trace_task
state, retval, uuid, args, kwargs, None,
File "/opt/ddr-local/ddrlocal/webui/tasks/files.py", line 157, in after_return
entity = Entity.from_identifier(Identifier(path=entity_path))
File "/opt/ddr-local/ddrlocal/webui/models.py", line 535, in from_identifier
return from_json(Entity, identifier.path_abs('json'), identifier)
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/models/common.py", line 811, in from_json
document.load_json(fileio.read_text(json_path))
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/models/entity.py", line 434, in load_json
json_data = common.load_json(self, module, json_text)
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/models/common.py", line 711, in load_json
list(f.values())[0]
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/modules.py", line 112, in function
value = function(value)
File "/opt/ddr-local/ddr-defs/repo_models/entity.py", line 1080, in jsonload_topics
converters.text_to_bracketids(text, ['term','id'])
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/vocab.py", line 699, in TEMP_scrub_topicdata
for term in get_vocabs(config.VOCABS_URL)['topics']['terms']
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/vocab.py", line 637, in get_vocabs
cached = _get_vocabs_all_http(vocabs_url)
File "/opt/ddr-local/venv/ddrlocal/lib/python3.7/site-packages/ddr_cmdln-0.9.4b0-py3.7.egg/DDR/vocab.py", line 609, in _get_vocabs_all_http
return_dict = multiprocessing.Manager().dict()
File "/usr/lib/python3.7/multiprocessing/context.py", line 56, in Manager
m.start()
File "/usr/lib/python3.7/multiprocessing/managers.py", line 543, in start
self._process.start()
File "/usr/lib/python3.7/multiprocessing/process.py", line 110, in start
'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
I've tried this in my dev environment, and in an install from tgz in a fresh Debian 10 Buster VM and haven't been able to duplicate this. I'm using Celery 4.4.2 which is the version range mentioned in the Celery issue mentioned above.
Can you try something for me? Add this code to a new file in /opt/ddr-local/ddrlocal/webui/apps.py
:
import os
from django.apps import AppConfig
from django.conf import settings
from DDR import vocab
class DDRLocalConfig(AppConfig):
name = 'webui'
verbose_name = "Densho Digital Repository Editor"
def ready(self):
if os.environ.get('RUN_MAIN'):
startup()
def startup():
# startup code goes here
vocab.get_vocabs(settings.VOCABS_URL)
Add this line to the top of /opt/ddr-local/ddrlocal/webui/__init__.py
default_app_config = 'webui.apps.DDRLocalConfig'
Restart the app using sudo service supervisor restart
and try the upload again.
Still getting the same error message when uploading binaries after making the modifications as suggested by jost in previous comment.
Could you try editing this line in /opt/ddr-local/Makefile
,
celery -A ddrlocal worker -l INFO -f /var/log/ddr/worker.log
adding --pool=threads
,
celery -A ddrlocal worker --pool=threads -l INFO -f /var/log/ddr/worker.log
and then running make runworker
as the ddr
user?
Should be fixed in ddr-local
commit 5fa58b8
for package ddrlocal-rc_5.0.0-rc4
.
fixed in 5.0.0rc4
daemonic processes are not allowed to have children errror when uploading an .mp4 file. operation also left a uncommitted lock file in the collection. tested on latest release of ddrlocal-rc on Entiat. error gets thrown for all media types. tiff mp3 mp4 pdf jpg
see link: https://github.com/celery/celery/issues/4525
daemonicissue.zip