Implement a dynamic caching layer that establishes watches utilizing a ServiceAccount

[everettraven]

As part of managing installed content and reconciling any changes made to the managed content, we need to configure watches on the resources managed for a given ClusterExtension. After #971, given a ClusterExtension and the content to be managed, we can configure clients for establishing watches using the ServiceAccount specified in the ClusterExtension.

While the exact implementation may vary, the currently proposed approach is as follows:

For each ClusterExtension:

• Uses a similar approach as the helm-operator-plugins client.RestConfigMapper to generate a rest.Config that uses a token from the ServiceAccount provided in ClusterExtension.Spec.ServiceAccount.Name
  • In order to support authentication token refreshing during a watch, we will need to do some further customization to the rest.Config resource. In order to achieve this, we will implement a custom http.RoundTripper that will set the Authorization HTTP header to a valid token retrieved from the “TokenGetter” object. This is a similar approach that is taken by client-go to utilize a BearerTokenFile. The custom http.RoundTripper we create could be a lighter weight wrapper around the default used by client-go that can be configured via the rest.Config.WrapTransport option that uses a transport.WrapperFunc.
• Configures a new controller-runtime cache.Cache with the generated rest.Config
  • Note: It may be nice to have a lightweight abstraction on top to simplify the logic within the ClusterExtension reconciler for creating new caches and establishing watches for the managed resources
• Uses the existing approach for establishing watches with the controller.Controller Watch() method
• Starts the cache
• When a ClusterExtension is deleted, it’s cache is stopped and removed

A mermaid graph to visualize the logical flow:

graph TD
    A(ClusterExtension)
    B(Content To Manage)
    C(Managed Content Cache)
    D(RestConfigMapper)
    E(rest.Config)
    F(cache.Cache)
    G(controller.Watch)
    J(ClusterExtension Reconciler)
    L(Watch)

    A -- Provided To --> C
    B -- Provided To --> C
    C -- Fetches Rest Config --> D
    D -- Creates for ServiceAccount --> E
    C -- Creates --> F
    E -- Provided To --> F
    F -- Provided To --> G
    G -- Creates --> L
    L -- Events Trigger --> J
    C -- Establishes Watch With --> G

Acceptance Criteria:

• A new Go library/type that provided a ClusterExtension and Managed Content can establish watches on managed resources using the ServiceAccount specified in the ClusterExtension spec
  • Will require #971 be completed first. Likely could utilize the implementation of a generic token fetcher from #972
• Unit tests

[joelanford]

Is it possible to wire up and use a controller-runtime cache-delegating client.Client per ClusterExtension, where we configure it with an http.Client with the custom http.RoundTripper?

[everettraven]

@joelanford I've updated the RFC and this issue to now reflect the ideal use of the existing controller-runtime caching layer APIs