search

joowani / kq

Kafka-based Job Queue for Python
http://kq.readthedocs.io
MIT License
571 stars 24 forks source link

Hardcoded processes=1 #2

Closed srgkm closed 8 years ago

srgkm commented 8 years ago

Hi!

Thank you for this package.

Why do you use MP module and hardcode number of workers (1) in a pool?

https://github.com/joowani/kq/blob/master/kq/worker.py#L257

joowani commented 8 years ago

No problem! Thank you for your interest.

I use it for enforcing timeouts on jobs (I spawn a child process to do carry out the job and if it hits the timeout threshold I simply terminate it). I chose processes instead of threads/signals for OS compatibility issues. The pool only has a single process to minimize resources (e.g. file descriptors), and no more than one is needed as a worker processes jobs sequentially.

Thanks Joohwan

joowani commented 8 years ago

Closing ticket. Let me know if you have anymore questions!