Add commonAnnotations to Cassandra metadata

[ClusterJan]

What is missing?

There should be a commonAnnotations field in the cassandra.metadata section of the ressource K8ssandraCluster.

Why do we need it?

Other operators in our Kubernetes cluster are interfering with the resources created by the cass-operator, and we need to add annotations to exclude these resources from the other operators.

Environment

• K8ssandra Operator version:

  v1.9.1

Anything else we need to know?: No.

[andrejshapal]

Looks very usefull for argocd.

[Miles-Garnsey]

From discussions internally and reviewing the conversations on discord, it looks like what is being asked for here is annotations for the statefulset, pods etc. coming from cass-operator.

I was originally going to implement this in k8ssandra-operator, but

1. Annotating the podTemplateSpec won't necessarily annotate the statefulSet itself.
2. There is a preference amongst the rest of the team for adding an AdditionalAnnotations field in the cass-operator CRD instead of doing this via the podTemplateeSpec.

I'll raise a ticket on cass-operator first, then expose this in k8ssandra-operator as a second step. Note that this won't change annotations on other resources created by k8ssandra-operator, but my understanding is that fields already exist to control annotations for the other resources.