Controller-Manager: controller-runtime magic 🪄 !

[pthomison]

@ejether this should be gtg for an 👁️ from you!

This one is a bigger change, so we can def leave it stashed for awhile for discussion.

This starts to consume the controller manager for actually handling our controller logic. This was basically pulled from kubebuilder, but AFAICT its the standard method for using the controller-runtime library.

Instead of handling the k8s-watch of the resource ourselves, we can just tell the manager that we want to control a given resource and it will handle all of the listing and watching. All we have to do is provide a "reconcile function" which will get executed for each NS on launch and any subsequent NS creates, updates, or deletes.

Additionally we get some OOTB goodness with a metrics server and health endpoint.

Things not 100% related to this change, but that we are getting by hoisting some of our logic up to the main func

• Shared K8S client, we will only need to auth/set up the client at the very beginning and then it will be re-used
• Shared NamespacesConfig, by hoisting it to the main func then adding it to a shared object, all reconciles will be able to reference it

This def leans into consuming the framework and is a bit more "go-y", but IMO its worth it for us to be able to drop all the low level watch logic from controller libraries.

LMK your thoughts and if you have any questions or concerns!! 🤘

[ejether]

Yeah, I'm for it. @sudermanjr suggesting using this to manage v1.metadata on any object and using the controller logic will enable that if we end up going that way.

Otherwise, there are enough changes here over my head I'm inclined to merge them all and sort it out later.