search

taskiq-python / taskiq

Distributed task queue with full async support
MIT License
866 stars 52 forks source link

Kiq a task in a subprocess ? #352

Closed realitix closed 2 months ago

realitix commented 3 months ago

Hello, I have a specific problem. I'm using a library that has memory leaks. Therefore, I am forced to fork to properly release the memory. Is it possible with Taskiq to launch a task by forking directly?

realitix commented 3 months ago

Ok I saw: https://github.com/taskiq-python/taskiq/blob/0.11.7/taskiq/cli/worker/run.py#L135

It would be great if we could set the executor class from cli arguments !

realitix commented 3 months ago

Not so simple.. We can't put a ProcessPool in a already forked process (taskiq workers are already forks): AssertionError: daemonic processes are not allowed to have children

realitix commented 2 months ago

I think I can do it with --max-tasks-per-child

realitix commented 2 months ago

Duplicate of #309 Fixed