daphne-eu / daphne

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines
Apache License 2.0
67 stars 62 forks source link

[#877] persist hwloc info in different pipelines #890

Closed philipportner closed 2 weeks ago

philipportner commented 3 weeks ago

This patch persists the hwloc topology info used in vectorized pipelines over the execution of multiple pipelines. This significantly reduces overhead in workloads that consint of multiple pipelines. With the tested script this patch reduced overall runtimes from ~6s to ~0.9s.

closes #877