catalogmetadata client adds extreme latency to reconciliation

[joelanford]

I've narrowed this latency down to the Bundles() call that is made for every single reconcile.

I added logging to the catalogmetadata.Client.Bundles() call to report start time and end time, and the function consistently takes around 30s, and it pegs a CPU.

This seems like a bug we should treat with a fairly high priority.

[bentito]

I added some timing data to the Bundles func too, documenting times for the various return points (https://gist.github.com/bentito/9fbf0d81354caa52121f5c4e294bd506). The fake catalog is just 1 entry I think? Would ~350µs (the longest test timing in 1 run (not scientific!) number items in real catalog # of calls to Bundle get close to 30s?

[joelanford]

I think that's probably an order of magnitude off still. I suspect that our fake catalog entry isn't as complex and varied as what exists in operatorhub in terms of what takes the most time (i.e. icons, olm.bundle.object properties).

[acornett21]

Are we talking about this function? Or something else?

[joelanford]

Yep, that's the one.

[joelanford]

I think there's a broad change that needs to be made to solve this. We should add a controller for Catalog objects that manages a local cache for each catalog that exists. This controller would be able to:

• fetch catalog contents as soon as a catalog is unpacked (not wait until a cluster extension needs to be reconciled)
• delete caches for catalogs that no longer exist (rather than leave them around forever)
• organize the cache in a way that optimizes the interaction from the ClusterExtension reconciler (for one, we could essentially index by package name because a ClusterExtension only needs information for spec.packageName from the catalogs.)

Separate, but related, I think the catalogmetadata wrapper types are adding some unnecessary abstraction, and we will be better off interacting directly with the FBC data model (e.g. use only operator-registry's declcfg package, to the extent possible)