search

AndreyVMarkelov / prom-bitbucket-exporter

Prometheus Exporter for Bitbucket
https://marketplace.atlassian.com/plugins/ru.andreymarkelov.atlas.plugins.prom-bitbucket-exporter/server/overview
BSD 2-Clause "Simplified" License
52 stars 21 forks source link

calculateUsersCount: high CPU usage on large instance #17

Closed audunroe closed 5 years ago

audunroe commented 5 years ago

We've been using prom-bitbucket-exporter to monitor test instances while going through Bitbucket data center validation for our app.. We then noticed fairly high CPU usage while the servers were completely idle.

For "large" user directories, it turns out ScheduledMetricEvaluatorImpl.calculateUsersCount is a fairly expensive operation. In our case, there's two directories (one LDAP one Crowd) with approx 10k users each. CPU usage sits at a more or less constant 25% on each of the four Bitbucket Datacenter nodes, even, when the servers are otherwise completely idle.

For now, I've simply disabled the entire executor (as we don't really need these particular metrics) which seems to have solved problem. But a proper fix would perhaps be if collection of additional metrics such as this (particularly metrics that can be costly) was configurable through the add-on.

Attached zip with jstack dump of the offending thread, and a flame graph illustrating the problem. FG was generated on a completely idle instance (no incoming user requests). It has some missing symbols due to insufficient warmup, and also some churn, but it was none the less enough to find the exact offending thread with jstack.

stack.zip

AndreyVMarkelov commented 5 years ago

Will do

AndreyVMarkelov commented 5 years ago

@audunroe Fixed. Thank you for reporting the bug.