Closed HorlogeSkynet closed 4 years ago
Since we're executing I/O-heavy work, should we use len(Entries)
(or maybe a static value, so 18 currently), as our ThreadPoolExecutor
's maxcount
? This would mean that in a case where nearly all entries were blocking on I/O, we'd still get at least another thread for another entry to begin its processing while waiting on the others - which should result in the quickest possible execution time. We don't really do anything heavy which is avoided by not spawning threads, and the improvements in Python 3.8 and above also mean we don't spawn more threads than ever necessary to complete all of the work in parallel.
You're actually right :+1: So what about mixing our needs with "new" Python 3.8+ behavior ?
executor = cm_stack.enter_context(
- ThreadPoolExecutor(max_workers=((os.cpu_count() or 1) * 5))
+ ThreadPoolExecutor(max_workers=min(len(enabled_entries), (os.cpu_count() or 1) + 4))
)
Reference : https://github.com/python/cpython/pull/13618
Nice catch ! Thanks :slightly_smiling_face:
This patch is mainly inspired from the work of @ingrinder (see
2bbc2dae
). You may notice an execution up to twice as fast.This behavior could be disabled with the new
parallel_loading
configuration option.See #68.
How has this been tested ?
Locally & test cases.
Types of changes :
Checklist :
[ ] [IF NEEDED] I have updated the test cases (which pass) accordingly ;