Annonation driven prometheus config

[suikast42]

Introduce platform specific tags (k/v) for generating grafana agent config.

Arcitecture.

• [ ] migrate the current config to file_sd_configs. That gives the agent the ability to manage dnyamic scrape config without restarts.
• [ ] Define and document the label structure for monitoring. Matric api path, Custom labels an so on.
• [ ] Implement a template that gerenetes the config from the platorm tags.

[suikast42]

Nats exporter for example is statically confugured.

Find a way to confugure exporters like db etc in a dynamic way like treafik tags.

For example see: https://medium.com/@jawaharsbs/practical-hashicorp-nomad-and-consul-monitoring-autoscaling-using-prometheus-grafana-part-3-8e032fbf1357

or https://admantium.medium.com/collecting-hardware-metrics-with-nomad-240beb6492c4

That could be strating point

{{- range services }}
  {{- range service .Name }}
   {{- if .Tags | contains "prometheus_exporter"}}
    {{ .Name}} : {{.Port}} {{.Tags }}
   {{- end}}
  {{- end}}
{{- end }}

Or look at dynamic consul_sd labeling from this approach here

https://github.com/cyriltovena/observability-nomad/tree/main/jobs

Use https://github.com/grafana/tns for testing metrics