Closed PhilippvK closed 4 months ago
I gave this a try using a benchmark I was performing recently.
process_pool, per_stage=1: not supported
process_pool, per_stage=0: 53s
thread_pool, per_stage=1: 15m22s
thread_pool, per_stage=0: 5m28s
I got a speed up of 5x over the ThreadPoolExecutor version on a 18C/36T machine which is great.
Visualization:
Scripts used to generate this plots can be found here: https://github.com/PhilippvK/mlonmcu-monitoring
This PR has been superseded by the feature-process-pool-new
Branch
See #153 for more context
Summary of changes:
Run.session
in favor of passing the session as argument when needed (Session class can not be serialized)Run.has_target()
helperMlonMcuContextMinimal
to pass environment settings and cache to processes (Locks can not be serialized)Limitations:
runs_per_stage
feature does not work together withprocess_pool
--progress
)Run.prefix
property is missing session idx/tmp
instead of session dirTODOs:
Run.__deepcopy__
as not needed anymore