rivosinc / prometheus-slurm-exporter

Export select slurm metrics to prometheus
Apache License 2.0
26 stars 5 forks source link

[exporter] expand all slurm arrays #66

Closed abhinavDhulipala closed 4 weeks ago

abhinavDhulipala commented 1 month ago

slurm arrays that are pending show up as one line. The -r option removes this optimization. Eventually, we can support parsing the array string, but there is quite a bit to the format and after a couple preliminary perf tests, the scrape duration seems unaffected. Will do some more perf testing before merging and will publish the dashboard pre and post change

abhinavDhulipala commented 1 month ago

The -r command doesn't seem to work with the json command. I'm switching the cli fallback to the default mode of operation for now until we get to supporting the array string.

abhinavDhulipala commented 1 month ago

So memory usage spiked by about 3X:

Screenshot 2024-06-05 at 12 02 36 AM

Our steady state before averaged 20 Mb, we now average 60 Mb on the heap and our max spiked from 60Mb to 125 Mb , respectively. This is undoubtedly high, but it's warranted for the sake of accuracy.

@kbroch-rivosinc Thoughts?

abhinavDhulipala commented 4 weeks ago

Tagging and releasing