dora-metrics / pelorus

Automate the measurement of organizational behavior
https://pelorus.readthedocs.io/
Apache License 2.0
245 stars 83 forks source link

Add performance measurement for collections #602

Closed mpryc closed 1 year ago

mpryc commented 2 years ago

As a result of slack conversation:

[kgranger]
 If performance is still a concern... maybe it's time we start measuring these things. Instrument the code, or at least just log start and end time / duration.

But also, since waiting to retrieve things from OpenShift can be really slow, we can combine the following things:
paginate requests
run those requests in a concurrent.futures.ThreadPoolExecutor and let the processing code handle pages as they come.
Since it's mostly waiting on IO, there won't be GIL contention.
I think this will become more relevant as people have more and more Builds and Images. Unfortunately, without telemetry, we won't know.

[mpryc]
great idea, definatelly for each "make_collector" we should have some numbers, would it make sense to store this in prometheus as well so we can always query on those?

[kgranger]
That makes sense to me!  In addition to logging it.
mpryc commented 1 year ago

Nice to have, we can always add more metrics around performance of exporters. Not really an issue.