Epic: spec/status on all CRDs and use standard controller patterns

[erikgb]

Most CRDs defined in trivy-operator are currently modelled as database schemas, not using best practices for Kubernetes API: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md. I understand that this was the goal in starboard/starboard-operator, but this is now trivy-operator - and I think it is about time to revisit this discussion.

IMO there are countless reasons for why we should follow Kubernetes API design best practices and implement controllers the way controllers are supposed to be designed. But the by far most important general reason to me, is to allow the machinery to operate safely in an efficient manner. Controllers are supposed to work declaratively and level-based, and generally not perform imperative operations as CREATE/DELETE. So controllers should in general do reconcile and use CreateOrUpdate for creating/updating resources and mostly rely on the Kubernetes garbage collection mechanisms to clean up.

There are workarounds implemented in trivy-operator now, that will just work out of the box with a proper design. And there are also open issues that are hard to fix without a clear distinction of the user intent (spec) from the controller feedback/state (status).

Re-modelling the CRDs will definitely be a breaking change, but I am convinced that it will pay back for the pain in the migration - with interests. My initiative and work on generating CRDs was mainly motivated by spec/status on CRDs as the end goals. And generating CRDs from code will surely ease the job of actually changing this in trivy-operator.

[chen-keinan]

Thanks for adding this issue , as discussed I'll set up a meeting so we could discuss on the solution and agree on how to move forward with it.