flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 50 forks source link

hostlist: perf issue in `hostlist_find_host()` due to `hostname_create()` #5987

Closed grondo closed 2 weeks ago

grondo commented 4 months ago

@trws noted in some perf benchmarking of Fluxion that hostlist_find_host() and specifically hostname_create() was taking a large percentage of time due mainly to the calloc(3) used in creating the struct hostname used for matching purposes.

Consider a strategy that doesn't require memory allocation to speed up repeated use of hostlist_find_host()

trws commented 4 months ago

Thanks for filing this @grondo! To add a bit of context, here's a slice of the perf output from the updated version of sched:

image

Overall it looks like hostlist ends up spending a good amount of time on allocations, and on repeatedly calculating lengths. Did we ever end up with a lib for string-with-length-and-capacity? It occurs to me because I'm working with the one git is using to fix the archive bug over there, and something like that would squash a lot of this and probably other similar things if we could easily pass those around.

grondo commented 2 weeks ago

I think this was addressed by #6259