bartongroup / slivka

http://bartongroup.github.io/slivka/
Apache License 2.0
7 stars 3 forks source link

Start local-queue jobs "asyncio" way #76

Closed warownia1 closed 4 years ago

warownia1 commented 4 years ago

Instead of using workers pool, utilize the fact that green threads are lightweight and can easily be spawned and disposed. Refer to comments in #72 .

Performance tests running 1M dummy jobs (sleep(0.0)) with 8 parallel workers

warownia1 commented 4 years ago

The advantage of having one worker per job makes it easier to cancel running jobs. The worker may catch CancelledError, terminate the sub-process, clean-up and die instead of looping forever. Additionally, failed workers will not deplete the pool and master thread is not needed restarting workers is not needed anymore.