search

stefanprodan / k8s-prom-hpa

Kubernetes Horizontal Pod Autoscaler with Prometheus custom metrics
MIT License
564 stars 192 forks source link

couldn't see container_* api metrics #9

Open jamyao opened 6 years ago

jamyao commented 6 years ago

Hello, I'm running kubernetes 1.10, and followed instruction here and installed metrics server, custom-metrics-api, and prometheus. I'm able to do autoscaling based on http_requests, but i can not do that based on container_network_receive_bytes_total, i noticed that httprequests metrics exposed at "/apis/custom.metrics.k8s.io/v1beta1", see output below: [xyz@master k8s-prom-hpa]$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|jq .|grep http_requests "name": "pods/http_requests_count", "name": "pods/http_requests", "name": "namespaces/http_requests", "name": "jobs.batch/http_requests", "name": "pods/http_requests_sum", "name": "jobs.batch/http_requests_bucket", "name": "jobs.batch/http_requests_count", "name": "jobs.batch/http_requests_sum", "name": "namespaces/http_requests_count", "name": "namespaces/http_requests_sum", "name": "pods/http_requests_bucket", "name": "namespaces/http_requestsbucket",

but this is the case for "container_network_receive_bytes_total"

_[xyz@master k8s-prom-hpa]$ kubectl get --raw container_network_receive_bytes_total|jq .|grep network "name": "pods/network_udp_usage", "name": "pods/network_tcpusage",

In Prometheus , i can see "container_network_receive_bytestotal" with data, actually all the metrics start with "container" in Prometheus didn't get exposed at /apis/custom.metrics.k8s.io/v1beta1..., any help is really appreciated

my system information: [xyz@master k8s-prom-hpa]$ kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

[xyz@master k8s-prom-hpa]$ kubectl api-versions admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 apps/v1beta1 apps/v1beta2 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 batch/v1 batch/v1beta1 certificates.k8s.io/v1beta1 custom.metrics.k8s.io/v1beta1 events.k8s.io/v1beta1 extensions/v1beta1 metrics.k8s.io/v1beta1 networking.k8s.io/v1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 stable.example.com/v1 storage.k8s.io/v1 storage.k8s.io/v1beta1 v1

[yaoky@master k8s-prom-hpa]$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default php-apache-8699449574-8hlqf 1/1 Running 0 3h default podinfo-65fc48f955-9xwdd 1/1 Running 0 22h default podinfo-65fc48f955-fnzx4 1/1 Running 0 22h default vtc1-deployment-6f864dbf54-lpq2d 1/1 Running 0 22h default vtc2-deployment-7f65859fc9-nbt2x 1/1 Running 0 22h kube-system etcd-master 1/1 Running 1 23h kube-system heapster-cbfb87c9c-sqknr 1/1 Running 0 23h kube-system kube-apiserver-master 1/1 Running 0 23h kube-system kube-controller-manager-master 1/1 Running 2 23h kube-system kube-dns-86f4d74b45-mgb44 3/3 Running 3 23h kube-system kube-proxy-52ctt 1/1 Running 0 23h kube-system kube-proxy-cgbb4 1/1 Running 1 23h kube-system kube-proxy-fw46f 1/1 Running 0 23h kube-system kube-proxy-rb4l5 1/1 Running 0 23h kube-system kube-scheduler-master 1/1 Running 2 23h kube-system metrics-server-6fbfb84cdd-cs7hk 1/1 Running 0 23h kube-system monitoring-grafana-69df66f668-87qkl 1/1 Running 0 23h kube-system monitoring-influxdb-78d4c6f5b6-jk59b 1/1 Running 0 23h kube-system weave-net-4qvhg 2/2 Running 0 23h kube-system weave-net-6mg8p 2/2 Running 0 23h kube-system weave-net-9bvqg 2/2 Running 0 23h kube-system weave-net-z9mmz 2/2 Running 3 23h monitoring custom-metrics-apiserver-7dd968d85-sw7fl 1/1 Running 0 22h monitoring prometheus-7dff795b9f-6jpd5 1/1 Running 0 22h

stefanprodan commented 6 years ago

The Prometheus adapter keeps only some metrics and does renaming on them, please take a look here: https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/format.md

jamyao commented 6 years ago

Thanks @stefanprodan . According to document you mentioned, "container_network_receive_bytes_total " suppose to be kept by Prometheus adapter, am i correct? it's not the case for me, i couldnt see this metrics

[xyz@master ~]$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|jq .|grep network "name": "pods/network_tcp_usage", "name": "pods/network_udp_usage",

Thanks