django worker slow?

[ghost]

I have a very simple task:

@job
def log_info():
    logger.info('Job running')

Queuing this task 500 times nearly takes 2 minutes to process with a single worker.

for i in range(500): tasks.log_info.delay()

I guess its because of the issue described in the Performance notes here. Is django-rq is setting up the django environment for every single job?

[zoidyzoidzoid]

Have you tried using a custom worker and seen if it improves performance?

[ghost]

Not sure if I came across custom workers, but pretty sure I haven't tried.

[zoidyzoidzoid]

Sorry, I used the wrong terminology.

The Performance notes that you linked to suggest a custom rqworker script, that is supposed to improve performance by preloading imports, so maybe setting up the django environment once.

I'm considering trying it since I have the same problem you're experiencing.

[jorgecarleitao]

I believe yes, django-rq is setting its env every call, like rq. You can easily test it by having a print('Hello') in settings.py. If print is being called after each job, then you have your answer.

I'm not sure this is an issue of django-rq though. As pointed out in rq docs that you link, it was a design decision to avoid memory leaks. Django-rq is backing it.

[satyrius]

@jorgecarleitao print will be once btw