Unable to get metrics in prometheus

[aelmehra]

Hello,

Ive followed all of the instructions here but I am still not able to get any of the container metrics in prometheus. I am returning lots of other metrics such ass process_cpu_seconds_total, apiserver_audit_event_total, and kubelet_docker_operations to name a few. Is there something that needs to be done prior to this?

[debianmaster]

Not sure how to debug your issue. but try seeing prometheus pod logs for any potential errors

[aelmehra]

No errors in the pods logs. Do I need to enable cluster metrics? Enabling Cluster Metrics

[toni-moreno]

I have cluster metrics enabled with heapster + hawkular

And I've created a new separated monitoring project ( instead of deploying on the openshift-infra ).

No errors in any of the containers, and can see some data on the prometheus console

but nothing in the grafana dashboard.

Can you tell me what to do to fix?

OpenShift Master: v3.7.1+282e43f-42 Kubernetes Master: v1.7.6+a08f5eeb62

[toni-moreno]

I've recreated on openshift-infra project but does not work either.

I can see that there is some metrics on the Prometheus frontend but there is not any "container*" metric.

[sleter]

I have exact same problem. I can't see all the metrics in Prometheus.

Openshift 3.9

[debianmaster]

@toni-moreno @sleter sorry i missed your message. how did you install prometheus on openshift?

better way is to use . ansible playbooks rather than configuring manually. like this

ansible-playbook openshift-ansible/playbooks/openshift-prometheus/config.yml

[debianmaster]

its pretty straight forward no tweaks needed.

[sleter]

@debianmaster Thanks for reply! I will give it a go

[aelmehra]

It looks like openshift-ansible/playbooks/openshift-prometheus/config.yml deploys the prometheus rules with most of the container_ metrics off

This is a excerpt from the configmap prometheus.yml

  # Exclude a set of high cardinality metrics that can contribute to significant
  # memory use in large clusters. These can be selectively enabled as necessary
  # for medium or small clusters.
  metric_relabel_configs:
  - source_labels: [__name__]
    action: drop
    regex: 'container_(cpu_user_seconds_total|cpu_cfs_periods_total|memory_usage_bytes|memory_swap|memory_working_set_bytes|memory_cache|last_seen|fs_(read_seconds_total|write_seconds_total|sector_(.*)|io_(.*)|reads_merged_total|writes_merged_total)|tasks_state|memory_failcnt|memory_failures_total|spec_memory_swap_limit_bytes|fs_(.*)_bytes_total|spec_(.*))'

After I changed the action from drop to keep all of the metrics showed up.

[sleter]

Thanks guys! I installed openshift-metrics and openshift-prometheus with ansible and now I can see all the metrics.