Allowlist of required metrics for federated setup

mihaitodor commented 3 years ago

We're using a federated Prometheus setup, where we're pulling metrics from various clusters via Prometheus federation and we're using the dashboards generated from this repo.

However, there is some concern that we'll get way too much data from the likes of kube-state-metrics, but we can't filter them by namespace, since, for example, kube_pod_container_resource_requests_cpu_cores is labelled using the namespace of the pod.

Ideally, we'd like to have an allowlist of the metrics that are used by these dashboards, but, given the current code structure, it's not easy to extract / generate this information programatically.

paulfantom commented 3 years ago

Sadly we don't have such a list at hand and going through all rules and dashboards to create it manually is not feasible and also such a list can quickly go stale.

I wonder if there is some way to extract metrics from generated grafana dashboards and prometheus rules. Sounds like such tool would be useful not only for this project but for the whole ecosystem.

mihaitodor commented 3 years ago

Thank you for the answer! Such a tool would be useful indeed, but I don't know if anything like that exists or if it can be built easily from existing libraries.

Another idea that came to mind was that maybe there can be a map of metrics and then each query references the metrics they need from that map instead of embedding them as plain strings. However, that will make the code more convoluted / harder to maintain...

paulfantom commented 3 years ago

Another idea that came to mind was that maybe there can be a map of metrics and then each query references the metrics they need from that map instead of embedding them as plain strings. However, that will make the code more convoluted / harder to maintain...

That would indeed make it harder and jsonnet is arguably already not very user-friendly.

Such a tool would be useful indeed, but I don't know if anything like that exists or if it can be built easily from existing libraries.

Prometheus somehow does it, so it may be possible ;)

hjet commented 3 years ago

Stumbled upon this issue by chance and just wanted to share a quick tool I wrote to do exactly this: mixin-metrics.

It's a bit hacky atm (uses gojq 🙃), but was meant to solve this problem temporarily while I integrate some features and fixes into cortex-tools — see the grafana-analyse and prometheus-analyse commands (h/t @gouthamve who wrote these).

Note:

mixin-metrics works on JSON/YAML output files
cortex-tools works directly against a Grafana instance and Prometheus instance, and currently only supports parsing metrics from dashboards (no rules yet). ~~It currently fails on the K8s mixin dashboards — see https://github.com/grafana/cortex-tools/pull/192#issuecomment-862389981.~~

In the short term (next month or so), you can expect the following to get added to cortex-tools:

rules parsing
JSON/YAML file input
prom relabel_config output
misc bugfixes
better docs
tests

Once that's in I will deprecate mixin-metrics but for now would use / adapt that code.

mihaitodor commented 3 years ago

Wow, that's awesome @hjet! I was looking at github.com/prometheus/prometheus/promql/parser, but didn't really got a chance to play with it. Was thinking that, at least for the Prometheus rules part, it might be a good addition to the promtool cli.

Your mixin-metrics utility does solve the issue I raised, so thanks a lot for building and sharing it!

paulfantom commented 3 years ago

@hjet awesome tool!

I just wrote a simple wrapper for kubernetes-mixin in form of following shell script (it removed all recording rule from the metrics output):

#!/bin/bash

mkdir -p tmp
cp prometheus_alerts.yaml prometheus_rules.yaml tmp/

M=$(mixin-metrics --dir=tmp rules --print | tr '|' '\n' | grep -v ':')
M="${M} $(mixin-metrics --dir=dashboards_out dash --print | tr '|' '\n' | grep -v ':')"

echo ${M} | tr ' ' '\n' | sort | uniq

And it printed out following metrics:

aggregator_unavailable_apiservice
aggregator_unavailable_apiservice_total
apiserver_client_certificate_expiration_seconds_bucket
apiserver_client_certificate_expiration_seconds_count
apiserver_request_duration_seconds_bucket
apiserver_request_duration_seconds_count
apiserver_request_terminations_total
apiserver_request_total
container_cpu_cfs_periods_total
container_cpu_cfs_throttled_periods_total
container_cpu_usage_seconds_total
container_fs_reads_bytes_total
container_fs_reads_total
container_fs_writes_bytes_total
container_fs_writes_total
container_memory_cache
container_memory_rss
container_memory_swap
container_memory_working_set_bytes
container_network_receive_bytes_total
container_network_receive_packets_dropped_total
container_network_receive_packets_total
container_network_transmit_bytes_total
container_network_transmit_packets_dropped_total
container_network_transmit_packets_total
go_goroutines
kube_daemonset_status_current_number_scheduled
kube_daemonset_status_desired_number_scheduled
kube_daemonset_status_number_available
kube_daemonset_status_number_misscheduled
kube_daemonset_updated_number_scheduled
kube_deployment_metadata_generation
kube_deployment_spec_replicas
kube_deployment_status_observed_generation
kube_deployment_status_replicas_available
kube_deployment_status_replicas_updated
kube_horizontalpodautoscaler_spec_max_replicas
kube_horizontalpodautoscaler_spec_min_replicas
kube_horizontalpodautoscaler_status_current_replicas
kube_horizontalpodautoscaler_status_desired_replicas
kube_job_failed
kube_job_spec_completions
kube_job_status_succeeded
kubelet_certificate_manager_client_expiration_renew_errors
kubelet_certificate_manager_client_ttl_seconds
kubelet_certificate_manager_server_ttl_seconds
kubelet_cgroup_manager_duration_seconds_bucket
kubelet_cgroup_manager_duration_seconds_count
kubelet_node_config_error
kubelet_node_name
kubelet_pleg_relist_duration_seconds_bucket
kubelet_pleg_relist_duration_seconds_count
kubelet_pleg_relist_interval_seconds_bucket
kubelet_pod_start_duration_seconds_count
kubelet_pod_worker_duration_seconds_bucket
kubelet_pod_worker_duration_seconds_count
kubelet_running_container_count
kubelet_running_containers
kubelet_running_pod_count
kubelet_running_pods
kubelet_runtime_operations_duration_seconds_bucket
kubelet_runtime_operations_errors_total
kubelet_runtime_operations_total
kubelet_server_expiration_renew_errors
kubelet_volume_stats_available_bytes
kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_inodes
kubelet_volume_stats_inodes_used
kube_node_spec_taint
kube_node_status_allocatable
kube_node_status_capacity
kube_node_status_condition
kube_persistentvolume_status_phase
kube_pod_container_resource_limits
kube_pod_container_resource_requests
kube_pod_container_status_restarts_total
kube_pod_container_status_waiting_reason
kube_pod_info
kube_pod_owner
kube_pod_status_phase
kubeproxy_network_programming_duration_seconds_bucket
kubeproxy_network_programming_duration_seconds_count
kubeproxy_sync_proxy_rules_duration_seconds_bucket
kubeproxy_sync_proxy_rules_duration_seconds_count
kube_replicaset_owner
kube_resourcequota
kubernetes_build_info
kube_statefulset_metadata_generation
kube_statefulset_replicas
kube_statefulset_status_current_revision
kube_statefulset_status_observed_generation
kube_statefulset_status_replicas
kube_statefulset_status_replicas_ready
kube_statefulset_status_replicas_updated
kube_statefulset_status_update_revision
namespace_workload_pod
node_cpu_seconds_total
node_memory_Buffers_bytes
node_memory_Cached_bytes
node_memory_MemAvailable_bytes
node_memory_MemFree_bytes
node_memory_MemTotal_bytes
node_memory_Slab_bytes
process_cpu_seconds_total
process_resident_memory_bytes
prometheus
rest_client_request_duration_seconds_bucket
rest_client_requests_total
scheduler_binding_duration_seconds_bucket
scheduler_binding_duration_seconds_count
scheduler_e2e_scheduling_duration_seconds_bucket
scheduler_e2e_scheduling_duration_seconds_count
scheduler_scheduling_algorithm_duration_seconds_bucket
scheduler_scheduling_algorithm_duration_seconds_count
scheduler_volume_scheduling_duration_seconds_bucket
scheduler_volume_scheduling_duration_seconds_count
storage_operation_duration_seconds_bucket
storage_operation_duration_seconds_count
storage_operation_errors_total
up
volume_manager_total_volumes
workqueue_adds_total
workqueue_depth
workqueue_queue_duration_seconds_bucket

It seems there are few false-positives like prometheus.

Either way, I think having such a list of metrics as a part of auto-generated doc would help some people, hence I am reopening this.

hjet commented 3 years ago

Yeah the cortex-tools updates will solve this false positive prometheus case (should be the only one).

This is a result of the hacky jq approach (specifically this line). Wanted to make sure I got the metrics referenced in template queries but have to exclude those where type is set to datasource (e.g. this). Using dashboard structs and doing some more intelligent deserializing will solve this. For now there is probably some fancy jq expression that would solve this case...

And yeah, interface improvements and tests are on deck as well. I would not use either cortex-tools or mixin-metrics in any sort of automated way until these things are implemented...I'll circle back on this issue with any updates

brancz commented 3 years ago

Could we maybe even have the checked-in version of this be a JSON list? Then people could use it from their jsonnet code (or anything else that supports json).

hjet commented 3 years ago

Yeah the default output is JSON and looks something like this:

{
  "metricsfiles": [
    {
      "filename": "/Users/coachjetha/Desktop/dash_load/dashboard.09ec8aa1e996d6ffcd6817bbaff4db1b.json",
      "metrics": [
        "apiserver_request:availability30d",
        "apiserver_request_total",
        "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile",
        "code_resource:apiserver_request_total:rate5m",
        "go_goroutines",
        "process_cpu_seconds_total",
        "process_resident_memory_bytes",
        "prometheus",
        "workqueue_adds_total",
        "workqueue_depth",
        "workqueue_queue_duration_seconds_bucket"
      ],
      "parse_errors": [
        "jq parsing expr failed: .dashboard.templating.list[]?.query.query"
      ]
    },
    {
      "filename": "/Users/coachjetha/Desktop/dash_load/dashboard.200ac8fdbfbb74b39aff88118e4d1c2c.json",
      "metrics": [
        "kube_pod_container_resource_limits",
        "kube_pod_container_resource_requests",
        "kube_pod_info",

...

So a quick and easy modification is to spit out an object containing all metrics, and augment those existing objects with type information (rules/dashboard), rule group / dashboard name, etc. And can generate this as part of CI or whatever.

Even better is to capture the dependencies between rules and their source metrics so one could filter only source metrics (for example if using Cortex you may want to evaluate rules there and ship only source metrics from Prometheus), being able to filter metrics per scrape target etc... but all these things depend on how much time I have on my hands this month haha

hjet commented 3 years ago

Update: should be ready EOW

hjet commented 3 years ago

Ok, have merged the functionality into cortex-tools, should be a lot more robust now. See https://github.com/grafana/cortex-tools#analyse-dashboard and https://github.com/grafana/cortex-tools#analyse-rule-file. You can see sample output in the README as well (scroll up a bit to analyse grafana and analyse ruler).

There is one more outstanding PR to get this working on the kubernetes-mixin dashboards: https://github.com/grafana-tools/sdk/pull/159. I've pinged the maintainer so once that gets merged, I'll update cortex-tools and post here again.

I think right now it fails on 3-4 dashboards due to that deserialization type error...

mihaitodor commented 3 years ago

That's awesome, thank you! BTW, you might want to go fmt the changes in grafana-tools/sdk#159.

hjet commented 3 years ago

That's awesome, thank you! BTW, you might want to go fmt the changes in grafana-tools/sdk#159.

Ah good catch, was going too fast. Too late 😭

Almost there... https://github.com/grafana/cortex-tools/pull/206

mihaitodor commented 3 years ago

No worries, I assumed it might trip up the linter, but I guess they don't have that set up. Glad it's almost sorted and thanks for all the work!

hjet commented 3 years ago

Yeah, I guess so. Usually my editor runs it, too.

Just tested the https://github.com/grafana/cortex-tools/pull/206 branch on the dashboards generated off of master and looks like we're good to go.

So if for whatever reason it takes a long time to get merged in, you can begin using the tool:

make cortextool
./cmd/cortextool/cortextool analyse dashboard /path/to/dashboards_out/*.json

hjet commented 3 years ago

Merged! If I find some time this weekend will look into getting this hooked up to CI

mihaitodor commented 3 years ago

Awesome! Really cool to have this functionality. Thanks again! :)

kubernetes-monitoring / kubernetes-mixin

Allowlist of required metrics for federated setup #620