Refactor operator

[benfiola]

Currently, there are a few things about the operator I don't like - even though it works (at least for my personal use case and the features my development environment test for).

Python to Golang

minio-operator-ext is written in python because I've been a python developer for about a decade and find the language comfortable. However, lately, I've been writing a lot of golang and I'm finding that I like it a whole lot more than python. I enjoy its pragmatism, its simplicity and its ability to compile (and cross-compile!) executables with ease. Plus, compiling my code into executables produces tiny docker images - and that's been great, shaving down image sizes by two orders of magnitude.

Finally, I read this comment on reddit. minio-operator-ext does not have this problem because I use an embedded spec stored in the status field as a source of truth. However, I do accept that this is a flaw in kopf and a refactor to golang would fundamentally move me off of the framework.

The goal here is to refactor the operator to golang. While I'd love to avoid boilerplate, there's a bit of momentum around using kubebuilder to generate project scaffolds (with added scaffold upgrade support as of 9/2023) - so I'll probably utilize kubebuilder so that this code-base will likely be familiar to anyone who has worked with golang controllers before.

Process optimizations

I personally think the operator can be improved as it pertains to resource discovery. I think that:

• The operator can better 'discover' existing Tenant resources in advance
• 'Discover' existing resources on each Tenant (e.g., enumerating the buckets, users, policies, etc. that exist)
• With the above information, it can better validate (- in advance) whether a tenant exists, is connectable, and whether a resource conflict might exist (e.g., trying to create a MinioBucket when someone might have externally created a bucket with the same name).
• (Optionally) Adopt validating webhook controllers to prevent the creation of uncreatable resources. I've been meaning to do this for a while - but there isn't a great development story around these (often requiring creating tunnels into the cluster to an external in-development webhook) - and I've never really liked that.

[benfiola]

For now, this effort will be contained in the go branch.