[Discussion] Synthetic monitoring checks

[danielloader]

First off, thanks for this chart! It's great to have a turnkey solution to onboarding a kubernetes cluster into grafana cloud. I also don't know where else to pose this question as discussions are disabled, nor did it make sense in the synthetic testing repos.

One hole in my monitoring stack is the synthetic checking - it's offered by grafana cloud but I have no way of cleanly automating adding and removing checks when I add/remove ingress backed applications to my cluster.

This chart uses prometheus CRDs for monitoring and scraping, and clearly that's not going to cover this usecase so I am asking what would be the best solution to handling this problem?

As far as I see it there's a couple options:

• Grafana Agent can monitor Ingress/Gateway objects for their FQDNs (based on adding a label to the objects maybe?) and then register them to synthetic tests.
  • This will end up like the nginx ingress situation with dozens of configuration flags and strings as annotations/labels. Not bad but not nice.
• A new controller that has a CRD for grafana synthetic checks that I can deploy as a boolean flagged option in my helm charts much like people wrap ServiceMonitor and PodMonitor objects in other charts.
  • Downside is ownership of another controller/CRD by the grafana team.
  • Flexible to have versioned CRD specs to match the upstream synthetic check API.

The worst solution but the only one I can manage myself right now is a ArgoCD PostSync hooked job container with the grafana api key and a curl command to add the check - it's not impossible just shifting the maintenance overhead into every single application I deploy.

Any thoughts are welcome as I'd like to integrate more tightly with grafana cloud from a kubernetes deployment!

[skl]

Nice idea! I've asked Synthetic Monitoring team for their thoughts on this.

[bentonam]

Adding to the discussion, I understand the reasoning behind the request. Wanted to add that the Grafana Terraform Provider does support synthetic monitoring checks. https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/synthetic_monitoring_check