[kgranger]
If performance is still a concern... maybe it's time we start measuring these things. Instrument the code, or at least just log start and end time / duration.
But also, since waiting to retrieve things from OpenShift can be really slow, we can combine the following things:
paginate requests
run those requests in a concurrent.futures.ThreadPoolExecutor and let the processing code handle pages as they come.
Since it's mostly waiting on IO, there won't be GIL contention.
I think this will become more relevant as people have more and more Builds and Images. Unfortunately, without telemetry, we won't know.
[mpryc]
great idea, definatelly for each "make_collector" we should have some numbers, would it make sense to store this in prometheus as well so we can always query on those?
[kgranger]
That makes sense to me! In addition to logging it.
As a result of slack conversation: