Closed warownia1 closed 4 years ago
The advantage of having one worker per job makes it easier to cancel running jobs. The worker may catch CancelledError, terminate the sub-process, clean-up and die instead of looping forever. Additionally, failed workers will not deplete the pool and master thread is not needed restarting workers is not needed anymore.
Instead of using workers pool, utilize the fact that green threads are lightweight and can easily be spawned and disposed. Refer to comments in #72 .
Performance tests running 1M dummy jobs (sleep(0.0)) with 8 parallel workers