Task which Triggers Multiprocessing

[diegofcoelho]

Is it possible to trigger a multiprocessing task? I tried but run it as a django management command makes it finish with no fail (and also no task performed) after a few seconds.

[diegofcoelho]

I am kinda answering my question, although not completely satisfied.

I manage to start a multithreading task with dango-chroniker by using the mysterious and undocumented from multiprocessing.pool import ThreadPool but still no success with true multiprocess.

[chrisspen]

I'm not sure what you mean. All chroniker jobs are run in separate threads by default.

On Apr 19, 2017 6:06 PM, "Diego F Coêlho" notifications@github.com wrote:

I am kinda answering my question, although not completely satisfied.

I manage to start a multithreading task with dango-chroniker by using the mysterious and undocumented from multiprocessing.pool import ThreadPool but still no success with true multiprocess.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/chrisspen/django-chroniker/issues/82#issuecomment-295465412, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHHl-E4GJ-6Q81qlCy990tChSED2I0rks5rxoVVgaJpZM4M2WvF .

[diegofcoelho]

I meant that I wanted to start a multiprocessing task using chroniker but the processes shuts after no activity is performed.

Although a multithread was possible using the library I described previsouly, it wasn't possible using the standard library and multiprocessing (true isolated processes) can't work anyhow..

[chrisspen]

You need to give me a little more detail about how you're "starting a multiprocessing task", because that could mean almost anything. Like I said, all jobs are run as a separate process, so you don't need to do any more work than just implement your Django management command. If you're using nohup or screen or some other method to launch a process, it might be happening. However, the chroniker job won't track those and will immediately return after the process launches.

[diegofcoelho]

I am afraid I can not provide a piece of working code as a perfect representation of what I was intending but following the examples on Python Docs, I could use the code below:

from multiprocessing import Pool
from django.core.management.base import BaseCommand
import time
import os

class Command(BaseCommand):
    @staticmethod
    def f(x):
        return x*x

    def handle(self, *args, **options):
        # start 4 worker processes
        with Pool(processes=4) as pool:

            # print "[0, 1, 4,..., 81]"
            print(pool.map(self.f, range(10)))

            print("For the moment, the pool remains available for more work")

        # exiting the 'with'-block has stopped the pool
        print("Now the pool is closed and no longer available")

Then the chroniker thread/process would stay up until the processes finished and even if we don't bother to get messages from the standalone process, it should be executed. The problem is that is doesn't.

The closest it gets can be seen in the code below:

from django.core.management.base import BaseCommand

from multiprocessing import Pool
from app.models import car

class Command(BaseCommand):

    def f(x):
        print(x)

    def handle(self, *args, **options):
        # start 4 worker processes
        with Pool(processes=1) as pool:
            l = list(car.objects.all())
            # print "[0, 1, 4,..., 81]"
            print(pool.map(self.f, l))

            print("For the moment, the pool remains available for more work")

        # exiting the 'with'-block has stopped the pool
        print("Now the pool is closed and no longer available")

But its execution triggers:

Traceback (most recent call last):
  File "\Python\Python35-32\lib\multiprocessing\process.py", line 249, in _boots
trap
    self.run()
  File "\Python\Python35-32\lib\multiprocessing\process.py", line 93, in run self._target(*self._args, **self._kwargs)
  File "\Python\Python35-32\lib\multiprocessing\pool.py", line 108, in worker task = get()
  File "\Python\Python35-32\lib\multiprocessing\queues.py", line 345, in get return ForkingPickler.loads(res)
  File "\app\management\commands\DjChMP.py", line 4, in <module>
    from app.models import car
  File "app\models.py", line 20, in <module>
    from django.contrib.auth.models import User
  File "\Python\Python35\site-packages\django\contrib\auth\models.py", line 4, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "\Python\Python35\site-packages\django\contrib\auth\base_user.py", line 52, in <module>
    class AbstractBaseUser(models.Model):
  File "\Python\Python35\site-packages\django\db\models\base.py", line 105, in __new__
    app_config = apps.get_containing_app_config(module)
  File "\Python\Python35\site-packages\django\apps\registry.py", line 237, in get_conta
ining_app_config
    self.check_apps_ready()
  File "\Python\Python35\site-packages\django\apps\registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

BTW, there are a few things on chroniker that bothers me a bit and would like to show you.. Should I open a issue for each or a all-in-one? With a some instructions I can try to cooperate with the project but I have no idea how a python environment setup for that would be..

[chrisspen]

Thanks for the detail. That's interesting. In general, I don't see why that wouldn't work. The error you're getting is because the new process is trying to access Django settings, but doesn't have the right path and/or Django settings module name set.

And by all means, feel free to open issues for any bugs or design issues you see.

The way I resolve bugs is to reproduce the error in a unittest, which would be called like:

export TESTNAME=.testSomeIssue; tox -e py35-django110

and then fix the issue and confirm the test passes. However, bugs involving multiprocessing can be tricky to fix because Django's test runner has a built-in transaction which only works from the main process, so trying to access any Django tables or settings in processes outside of that will get you strange errors like the one you reported.