search

nilp0inter / threadedprocess

ThreadedProcessPoolExecutor implementation for Python
Other
8 stars 2 forks source link

TODO in _ThreadPoolExecutor._adjust_thread_count #3

Open bionicles opened 4 years ago

bionicles commented 4 years ago

to close the TODO in _ThreadPoolExecutor._adjust_thread_count

with self._available_thread._value -- that semaphore counts the number of idle threads, but the semaphore counts from max_workers, so if there are 3 / 5 workers running then the semaphore value reads 2 while there are no idle threads, so we gotta also subtract out potential threads

potentially_idle_threads = self._available_thread.value unused_threads = self.max_workers - num_threads idle_threads = potentially_idle_threads - unused_threads

then

threads_needed = idle_threads < self._work_queue.qsize()

another badass upgrade here would be for it to autoscale down if workload goes down ( if idle_threads > queue size for extended periods ) .. that might automatically happen

nilp0inter commented 4 years ago

Feel free to open a pull request and we can discuss over it :)