Improve observability metrics and dashboards [epic]

[maleck13]

What

Add a specific dashboard that shows important information from an engineering (platform and us) perspective around what the DNS operator is doing.

• Requests to external DNS providers (success, ratelimited etc) (instrumented client)
• State of a given dns record (IE is it ready/ still trying to be validated/ is it in a conflicted state etc)
• Number of reconciles for a given DNSRecord or Managed zone (controller runtime metrics)
• Total number of DNSRecords and Managed Zones in a given cluster

Other important things

https://github.com/kubernetes/kube-state-metrics/tree/main/docs#exposed-metrics

Done

Feel free to split these out into separate issues

https://github.com/Kuadrant/kuadrant-operator/issues/649

• [x] #206
• [ ] Expose kube state metrics
• [ ] Dashboard resource provided to be able to import this dashboard into a grafana instance
• [ ] Documentation providing details on the new metrics and how to load up the dashboard

[philbrookes]

related: https://github.com/Kuadrant/dns-operator/issues/206