State-of-world reconciliation

[guicassolato]

Enhance the controller package with optional state-of-world reconciliation:

• Alternative to incremental events only (provided today by the informers)
• Based on controller-runtime
• Inspired by Envoy Gateway

There are several good reasons to use controller-runtime, especially in situations where the event stream is unreliable (e.g. event loss; despite periodic resyncs implemented by the informers.) These reasons include:

• Event queue management (skipping, de-duplication, retries)
• Built-in metrics and operator health checks
• Rest API schema mapper
• Helpers to manage finalizers and owner references
• Testing tools (e.g. fake client)
• Admission webhooks service

Traps to avoid / recommendation:

• controller-runtime is famously bad at dealing with concurrency, hence taking inspiration on EG's approach (i.e. one single reconciler + watchable for async function calls)
• keep leveraging reconciliation 'workflow' and 'event subscription' models if possible

[guicassolato]

Proposal for an execution plan, based on the current code:

1. Replace cache with a watchable map
2. Define a Runnable interface: { Run(stopCh <-chan struct{}) }
3. Modify the list of informers of the controller to be a list of generic runnables
4. Modify the controller builder option WithInformer to WithRunnable
5. Add an informer builder option InformerBuilder(builder)
   • IncrementalInformer → returns a cache.SharedInformer that adds incremental changes directly to the cache (default informer builder)
   • StateReconciler → returns a runnable that simply adds a ListFunc to a slice of state reconciler ListFuncs of the controller
6. Modify controller.For to controller.Watch
   • Build the informer using the informer builder specified in the options
7. Add a controller builder option ManagedBy(manager)
8. Implement controller-runtime’s Reconciler interface
   • List from the cluster all resources whose kinds the controller is watching, by triggering all the state reconciler ListFuncs registered with the controller
   • Update the cache atomically with the newly fetched list of resources
9. Modify controller.Start():
   • Run all the runnables
   • If a manager was provided, create a controller-runtime controller, passing the manager and the PM controller as the reconciler:

     controller.New("gatewayapi", c.manager, controller.Options{Reconciler: c})
     c.manager.Start()

10. Rename CallbackFunc as ReconcileFunc
11. Modify the ReconcileFunc to receive a list of ResourceEvents
12. Modify controller builder option WithCallback to WithReconciler(ReconcileFunc)
    • Subscribe to the cache map
    • On messages published to the subscription channel:

      if snapshot, ok := <-subscription; ok {
      events := lo.Map(snapshot.Updates, …)
      topology := c.topology.Build(snapshot.State)
      reconcileFunc(ctx, events, topology)
      }