Consider Kibana resource watcher

[anyasabo]

We should consider creating a watcher tool for Kibana, that can watch config maps (or some other type of resource) and automatically import them into Kibana. This would let users have the same set of Kibana resources across environments, and allow for easy distribution (say, we could supply some default dashboards to observe the operator's metrics). My thinking is to initially support Kibana dashboards, but potentially expand it to Watcher watches. There may be more resources.

It's not obvious to me where this would live -- I'm leaning towards the operator, though it may make sense for it to be a sidecar in Kibana too. I think the downside is that you're running a container for replica when we really only need to import objects once per Kibana

Notes

The kibana team recommended we use the saved objects import/export API: https://www.elastic.co/guide/en/kibana/current/saved-objects-api-import.html

As the import/export dashboard API is being removed in 8.0: https://github.com/elastic/kibana/issues/41439

Examples in other projects

Grafana supports declaring them in config files that are loaded on startup: https://grafana.com/docs/administration/provisioning/#dashboards

which is how kube-prometheus project does it: https://github.com/coreos/kube-prometheus/blob/master/manifests/grafana-deployment.yaml#L42

https://github.com/coreos/kube-prometheus/blob/master/docs/developing-prometheus-rules-and-grafana-dashboards.md#dashboards

though there are projects using techniques like described here (and what kube-prom used to do) with a watcher for config maps: https://github.com/PierreVincent/k8s-grafana-watcher

[mtparet]

It looks like CRDs ! :)

[jbnjohnathan]

To include a KibanaDashboard or similar CRD would be very nice. Is there any progress to mention on this ticket yet?

[captainpro-eng]

+1