daphne-eu / daphne

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines
Apache License 2.0
67 stars 62 forks source link

--vec overhead because of poor hwloc usage #877

Closed philipportner closed 2 weeks ago

philipportner commented 4 weeks ago

Currently, when running DAPHNE with the --vec flag, the usage of hwloc_topology_load() inside of MTWrapperBase causes a significant overhead in execution time as this is repeated for every single call to a _vectorizedPipeline__* kernel. This causes us to repeatedly read sysfs files which each needs multiple syscalls.

Possible solutions could be:

philipportner commented 4 weeks ago

This is part of the problem that causes a script that runs in ~0.25s without --vec to explode up to ~6.2s with --vec