search

kubernetes-sigs / prometheus-adapter

An implementation of the custom.metrics.k8s.io API using Prometheus
Apache License 2.0
1.92k stars 554 forks source link

adapter unable to get Flink custom metric #182

Closed ghost closed 3 years ago

ghost commented 5 years ago

Hi,

I'm unable to get Flink custom metric that is being viewed in Promethues and Grafhana. When doing the same for Kafka it seems to be working.

Please advise,

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "jobs.batch/flink_taskmanager_job_task_operator_fomatterA_validate_success_rate",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

ConfigMap

{{- if not .Values.rules.existing -}}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "k8s-prometheus-adapter.fullname" . }}
  labels:
    app: {{ template "k8s-prometheus-adapter.name" . }}
    chart: {{ template "k8s-prometheus-adapter.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  config.yaml: |
    rules:
{{- if .Values.rules.default }}
    - seriesQuery: 'flink_taskmanager_job_task_operator_fomatterA_validate_success_rate{job="flink"}'
      resources:
        template: "<<.Resource>>"
      name:
        matches: ""
        as: ""
      #metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
{{- end -}}
{{- if .Values.rules.custom }}
{{ toYaml .Values.rules.custom | indent 4 }}
{{- end -}}
{{- end -}}

Adaptor logs:


I0326 08:30:19.073995       1 wrap.go:42] GET /healthz: (147.641µs) 200 [[kube-probe/1.13] 10.47.64.0:58036]
I0326 08:30:19.588869       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I0326 08:30:19.588992       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:19.590193       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:30:19.590212       1 round_trippers.go:411] Response Headers:
I0326 08:30:19.590219       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:19.590227       1 round_trippers.go:414]     Content-Length: 533
I0326 08:30:19.590235       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:19 GMT
I0326 08:30:19.590266       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I0326 08:30:19.590377       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:30:19.590476       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.002721ms) 200 [[kube-controller-manager/v1.13.3 (linux/amd64) kubernetes/721bfa7/system:serviceaccount:kube-system:resourcequota-controller] 10.47.0.0:43324]
I0326 08:30:20.773363       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:30:20.773477       1 round_trippers.go:386] curl -k -v -XPOST  -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" -H "Content-Type: application/json" -H "Accept: application/json, */*" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:20.774718       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:30:20.774744       1 round_trippers.go:411] Response Headers:
I0326 08:30:20.774752       1 round_trippers.go:414]     Content-Length: 406
I0326 08:30:20.774759       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:20 GMT
I0326 08:30:20.774769       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:20.774789       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:unauthenticated\""}}
I0326 08:30:20.774869       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:20.774894       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:20.774928       1 wrap.go:42] GET /healthz: (1.737947ms) 200 [[kube-probe/1.13] 10.47.64.0:58136]
I0326 08:30:23.983139       1 api.go:74] GET http://prom-helm-prometheus-server:80/api/v1/series?match%5B%5D=flink_taskmanager_job_task_operator_fomatterA_validate_success_rate%7Bjob%3D%22flink%22%7D&start=1553587823.982 200 OK
I0326 08:30:23.983249       1 api.go:93] Response Body: {"status":"success","data":[{"__name__":"flink_taskmanager_job_task_operator_fomatterA_validate_success_rate","host":"flink_helm_flink_taskmanager_5667667b76_rt7jv","instance":"10.39.64.25:9250","job":"flink","job_id":"c7a24a6a63b15bd29d8c1efa0f3cda0b","job_name":"formatterA","operator_id":"77967853ff3e29cd94e86ac64d38a63b","operator_name":"Process:_Message","subtask_index":"0","task_attempt_id":"0ca06624eaeed4e7cb97441921b71a5b","task_attempt_num":"0","task_id":"e3dfc0d7e9ecd8a43f85f0b68ebf3b80","task_name":"Source:_RECEIVER_A_OUT_____Process:_Call_____Map____Sink:__INT_MSG__Map____Sink:__ERROR___Process:_Message_____Map____Sink:_INT_MSG__Map____Sink:_ERROR__","tm_id":"dbbde9b46d5423bbd046184c0c00f10d"}]}
I0326 08:30:23.983394       1 provider.go:269] Set available metric list from Prometheus to: [[{flink_taskmanager_job_task_operator_fomatterA_validate_success_rate {host="flink_helm_flink_taskmanager_5667667b76_rt7jv", instance="10.39.64.25:9250", job="flink", job_id="c7a24a6a63b15bd29d8c1efa0f3cda0b", job_name="formatterA", operator_id="77967853ff3e29cd94e86ac64d38a63b", operator_name="Process:_Message", subtask_index="0", task_attempt_id="0ca06624eaeed4e7cb97441921b71a5b", task_attempt_num="0", task_id="e3dfc0d7e9ecd8a43f85f0b68ebf3b80", task_name="Source:_RECEIVER_A_OUT_____Process:_Call_____Map____Sink:__INT_MSG__Map____Sink:__ERROR___Process:_Message_____Map____Sink:_INT_MSG__Map____Sink:_ERROR__", tm_id="dbbde9b46d5423bbd046184c0c00f10d"}}]]
I0326 08:30:23.983525       1 resource_converter.go:170] unable to normalize group-resource task_attempt_num from label "task_attempt_num", skipping: no matches for {  task_attempt_num}
I0326 08:30:23.983564       1 resource_converter.go:170] unable to normalize group-resource instance from label "instance", skipping: no matches for {  instance}
I0326 08:30:23.983597       1 resource_converter.go:170] unable to normalize group-resource job_id from label "job_id", skipping: no matches for {  job_id}
I0326 08:30:23.983631       1 resource_converter.go:170] unable to normalize group-resource tm_id from label "tm_id", skipping: no matches for {  tm_id}
I0326 08:30:23.983692       1 resource_converter.go:170] unable to normalize group-resource host from label "host", skipping: no matches for {  host}
I0326 08:30:23.983728       1 resource_converter.go:170] unable to normalize group-resource operator_id from label "operator_id", skipping: no matches for {  operator_id}
I0326 08:30:23.983758       1 resource_converter.go:170] unable to normalize group-resource subtask_index from label "subtask_index", skipping: no matches for {  subtask_index}
I0326 08:30:23.983787       1 resource_converter.go:170] unable to normalize group-resource task_id from label "task_id", skipping: no matches for {  task_id}
I0326 08:30:23.983815       1 resource_converter.go:170] unable to normalize group-resource job_name from label "job_name", skipping: no matches for {  job_name}
I0326 08:30:23.983845       1 resource_converter.go:170] unable to normalize group-resource operator_name from label "operator_name", skipping: no matches for {  operator_name}
I0326 08:30:23.983875       1 resource_converter.go:170] unable to normalize group-resource task_attempt_id from label "task_attempt_id", skipping: no matches for {  task_attempt_id}
I0326 08:30:23.983904       1 resource_converter.go:170] unable to normalize group-resource task_name from label "task_name", skipping: no matches for {  task_name}
I0326 08:30:29.073867       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:29.073898       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:29.073925       1 wrap.go:42] GET /healthz: (152.992µs) 200 [[kube-probe/1.13] 10.47.64.0:58608]
I0326 08:30:30.772455       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:30.772489       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:30.772526       1 wrap.go:42] GET /healthz: (193.021µs) 200 [[kube-probe/1.13] 10.47.64.0:58698]
I0326 08:30:37.613928       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:kube-controller-manager","group":["system:authenticated"]},"status":{"allowed":false}}
I0326 08:30:37.614088       1 round_trippers.go:386] curl -k -v -XPOST  -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:37.617563       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 3 milliseconds
I0326 08:30:37.617591       1 round_trippers.go:411] Response Headers:
I0326 08:30:37.617599       1 round_trippers.go:414]     Content-Length: 443
I0326 08:30:37.617606       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:37 GMT
I0326 08:30:37.617613       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:37.617642       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:kube-controller-manager","group":["system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I0326 08:30:37.617787       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:30:37.617903       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (4.273704ms) 200 [[kube-controller-manager/v1.13.3 (linux/amd64) kubernetes/721bfa7/controller-discovery] 10.47.0.0:43324]
I0326 08:30:39.073113       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:30:39.073251       1 round_trippers.go:386] curl -k -v -XPOST  -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" -H "Accept: application/json, */*" -H "Content-Type: application/json" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:39.074637       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:30:39.074676       1 round_trippers.go:411] Response Headers:
I0326 08:30:39.074684       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:39.074691       1 round_trippers.go:414]     Content-Length: 406
I0326 08:30:39.074697       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:39 GMT
I0326 08:30:39.074717       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:unauthenticated\""}}
I0326 08:30:39.074806       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:39.074831       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:39.074871       1 wrap.go:42] GET /healthz: (1.870369ms) 200 [[kube-probe/1.13] 10.47.64.0:59174]
I0326 08:30:40.455474       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:30:40.455599       1 round_trippers.go:386] curl -k -v -XPOST  -H "Content-Type: application/json" -H "Accept: application/json, */*" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:40.456842       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:30:40.456868       1 round_trippers.go:411] Response Headers:
I0326 08:30:40.456876       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:40.456882       1 round_trippers.go:414]     Content-Length: 260
I0326 08:30:40.456898       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:40 GMT
I0326 08:30:40.456928       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:30:40.456997       1 authorization.go:73] Forbidden: "/", Reason: ""
I0326 08:30:40.457086       1 wrap.go:42] GET /: (1.734927ms) 403 [[Go-http-client/2.0] 10.47.0.0:43306]
I0326 08:30:40.771756       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:40.771784       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:40.771820       1 wrap.go:42] GET /healthz: (178.112µs) 200 [[kube-probe/1.13] 10.47.64.0:59278]
I0326 08:30:44.839686       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:30:44.839829       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (412.264µs) 200 [[kube-apiserver/v1.13.3 (linux/amd64) kubernetes/721bfa7] 10.47.0.0:43324]
I0326 08:30:47.067140       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:generic-garbage-collector","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I0326 08:30:47.067288       1 round_trippers.go:386] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:47.068505       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:30:47.068523       1 round_trippers.go:411] Response Headers:
I0326 08:30:47.068531       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:47.068537       1 round_trippers.go:414]     Content-Length: 534
I0326 08:30:47.068544       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:47 GMT
I0326 08:30:47.068577       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:generic-garbage-collector","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I0326 08:30:47.068703       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:30:47.068810       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (1.98223ms) 200 [[kube-controller-manager/v1.13.3 (linux/amd64) kubernetes/721bfa7/system:serviceaccount:kube-system:generic-garbage-collector] 10.47.0.0:43324]
I0326 08:30:49.073306       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:49.073341       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:49.073370       1 wrap.go:42] GET /healthz: (157.982µs) 200 [[kube-probe/1.13] 10.47.64.0:59758]
I0326 08:30:49.692749       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I0326 08:30:49.692891       1 round_trippers.go:386] curl -k -v -XPOST  -H "Content-Type: application/json" -H "Accept: application/json, */*" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:49.696563       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 3 milliseconds
I0326 08:30:49.696597       1 round_trippers.go:411] Response Headers:
I0326 08:30:49.696604       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:49.696611       1 round_trippers.go:414]     Content-Length: 533
I0326 08:30:49.696616       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:49 GMT
I0326 08:30:49.696645       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I0326 08:30:49.696760       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:30:49.696895       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (4.441445ms) 200 [[kube-controller-manager/v1.13.3 (linux/amd64) kubernetes/721bfa7/system:serviceaccount:kube-system:resourcequota-controller] 10.47.0.0:43324]
I0326 08:30:50.772099       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:30:50.772257       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:30:50.773752       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:30:50.773770       1 round_trippers.go:411] Response Headers:
I0326 08:30:50.773778       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:30:50.773797       1 round_trippers.go:414]     Content-Length: 406
I0326 08:30:50.773816       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:30:50 GMT
I0326 08:30:50.773839       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:unauthenticated\""}}
I0326 08:30:50.773935       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:50.773958       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:50.774000       1 wrap.go:42] GET /healthz: (2.02862ms) 200 [[kube-probe/1.13] 10.47.64.0:59856]
I0326 08:30:59.077855       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:30:59.077886       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:30:59.077922       1 wrap.go:42] GET /healthz: (164.801µs) 200 [[kube-probe/1.13] 10.47.64.0:60338]
I0326 08:31:00.772938       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:31:00.772967       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:31:00.772996       1 wrap.go:42] GET /healthz: (153.491µs) 200 [[kube-probe/1.13] 10.47.64.0:60438]
I0326 08:31:07.717676       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:kube-controller-manager","group":["system:authenticated"]},"status":{"allowed":false}}
I0326 08:31:07.717828       1 round_trippers.go:386] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" -H "Accept: application/json, */*" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:31:07.721420       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 3 milliseconds
I0326 08:31:07.721449       1 round_trippers.go:411] Response Headers:
I0326 08:31:07.721458       1 round_trippers.go:414]     Content-Length: 443
I0326 08:31:07.721465       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:31:07 GMT
I0326 08:31:07.721474       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:31:07.721510       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:kube-controller-manager","group":["system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I0326 08:31:07.721606       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:31:07.721787       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (4.426835ms) 200 [[kube-controller-manager/v1.13.3 (linux/amd64) kubernetes/721bfa7/controller-discovery] 10.47.0.0:43324]
I0326 08:31:09.073367       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:31:09.073507       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:31:09.074907       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:31:09.074933       1 round_trippers.go:411] Response Headers:
I0326 08:31:09.074952       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:31:09.074959       1 round_trippers.go:414]     Content-Length: 406
I0326 08:31:09.074974       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:31:09 GMT
I0326 08:31:09.074999       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:unauthenticated\""}}
I0326 08:31:09.075097       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:31:09.075122       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:31:09.075165       1 wrap.go:42] GET /healthz: (1.92485ms) 200 [[kube-probe/1.13] 10.47.64.0:60932]
I0326 08:31:10.456871       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:31:10.457017       1 round_trippers.go:386] curl -k -v -XPOST  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Content-Type: application/json" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:31:10.458525       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:31:10.458652       1 round_trippers.go:411] Response Headers:
I0326 08:31:10.458842       1 round_trippers.go:414]     Content-Length: 260
I0326 08:31:10.458962       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:31:10 GMT
I0326 08:31:10.459077       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:31:10.459205       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:31:10.459615       1 authorization.go:73] Forbidden: "/", Reason: ""
I0326 08:31:10.460069       1 wrap.go:42] GET /: (3.285123ms) 403 [[Go-http-client/2.0] 10.47.0.0:43306]
I0326 08:31:10.771669       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:31:10.771695       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:31:10.771730       1 wrap.go:42] GET /healthz: (185.631µs) 200 [[kube-probe/1.13] 10.47.64.0:32796]
I0326 08:31:14.852548       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:31:14.852709       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (379.993µs) 200 [[kube-apiserver/v1.13.3 (linux/amd64) kubernetes/721bfa7] 10.47.0.0:43324]
I0326 08:31:17.164940       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:generic-garbage-collector","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I0326 08:31:17.165079       1 round_trippers.go:386] curl -k -v -XPOST  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:31:17.166251       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:31:17.166269       1 round_trippers.go:411] Response Headers:
I0326 08:31:17.166277       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:31:17.166284       1 round_trippers.go:414]     Content-Length: 534
I0326 08:31:17.166291       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:31:17 GMT
I0326 08:31:17.166314       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:generic-garbage-collector","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I0326 08:31:17.166400       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:31:17.166557       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (1.90604ms) 200 [[kube-controller-manager/v1.13.3 (linux/amd64) kubernetes/721bfa7/system:serviceaccount:kube-system:generic-garbage-collector] 10.47.0.0:43324]
I0326 08:31:19.073236       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:31:19.073265       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:31:19.073293       1 wrap.go:42] GET /healthz: (152.652µs) 200 [[kube-probe/1.13] 10.47.64.0:33272]
I0326 08:31:19.799305       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I0326 08:31:19.799473       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:31:19.804075       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 4 milliseconds
I0326 08:31:19.804100       1 round_trippers.go:411] Response Headers:
I0326 08:31:19.804106       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:31:19.804112       1 round_trippers.go:414]     Content-Length: 533
I0326 08:31:19.804120       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:31:19 GMT
I0326 08:31:19.804143       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I0326 08:31:19.804228       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0326 08:31:19.804333       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (5.340955ms) 200 [[kube-controller-manager/v1.13.3 (linux/amd64) kubernetes/721bfa7/system:serviceaccount:kube-system:resourcequota-controller] 10.47.0.0:43324]
I0326 08:31:20.772060       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0326 08:31:20.772213       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb20tYWRhLXByb21ldGhldXMtYWRhcHRlci10b2tlbi1namd3NCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxYzc2ZmE1ZC00ZmExLTExZTktYTcxYi0wNmNkMjhhYTdmNDYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwcm9tLWFkYS1wcm9tZXRoZXVzLWFkYXB0ZXIifQ.JCYbcXXuC3CQxdk88SA-JcP7oCY_H2_zhOyAiUdfQ9C0Gori_m4Dq4Bd4dw-yeALmxIYH16CwqzW8RLQpun_5Ixb4kpTh9YIpPk0S5ZCzMFs4qFpBPLo_DPnm7Opu9M225gGDzVHwN0Jq-ZYft0Tui-5bD_eFJzvwSmKe4iidWZP6X_aDOwI0rnXfY8LpDesCwgELFyLJp3nasFZMhfxJ6qpRTb3pgwBzEoeVn7KC6nhL5RJ4MIP6Q3wawAr8FY6D0FJq-Y5cciYfhryX9G5LU2_bW_yqKUiax0jGBZ57wiZxupLjOgmSBvL5g3qCtP_m4urFxbBrffv1-BMwgNL1g" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0326 08:31:20.773575       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0326 08:31:20.773602       1 round_trippers.go:411] Response Headers:
I0326 08:31:20.773611       1 round_trippers.go:414]     Content-Type: application/json
I0326 08:31:20.773618       1 round_trippers.go:414]     Content-Length: 406
I0326 08:31:20.773624       1 round_trippers.go:414]     Date: Tue, 26 Mar 2019 08:31:20 GMT
I0326 08:31:20.773665       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:unauthenticated\""}}
I0326 08:31:20.773759       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0326 08:31:20.773783       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0326 08:31:20.773820       1 wrap.go:42] GET /healthz: (1.9197ms) 200 [[kube-probe/1.13] 10.47.64.0:33370]
I0326 08:31:23.985084       1 api.go:74] GET http://prom-helm-prometheus-server:80/api/v1/series?match%5B%5D=flink_taskmanager_job_task_operator_fomatterA_validate_success_rate%7Bjob%3D%22flink%22%7D&start=1553587883.984 200 OK
I0326 08:31:23.985196       1 api.go:93] Response Body: {"status":"success","data":[{"__name__":"flink_taskmanager_job_task_operator_fomatterA_validate_success_rate","host":"flink_helm_flink_taskmanager_5667667b76_rt7jv","instance":"10.39.64.25:9250","job":"flink","job_id":"c7a24a6a63b15bd29d8c1efa0f3cda0b","job_name":"formatterA","operator_id":"77967853ff3e29cd94e86ac64d38a63b","operator_name":"Process:_Message","subtask_index":"0","task_attempt_id":"0ca06624eaeed4e7cb97441921b71a5b","task_attempt_num":"0","task_id":"e3dfc0d7e9ecd8a43f85f0b68ebf3b80","task_name":"Source:_RECEIVER_A_OUT_____Process:_Call_____Map____Sink:__INT_MSG__Map____Sink:__ERROR___Process:_Message_____Map____Sink:_INT_MSG__Map____Sink:_ERROR__","tm_id":"dbbde9b46d5423bbd046184c0c00f10d"}]}
I0326 08:31:23.985336       1 provider.go:269] Set available metric list from Prometheus to: [[{flink_taskmanager_job_task_operator_fomatterA_validate_success_rate {host="flink_helm_flink_taskmanager_5667667b76_rt7jv", instance="10.39.64.25:9250", job="flink", job_id="c7a24a6a63b15bd29d8c1efa0f3cda0b", job_name="formatterA", operator_id="77967853ff3e29cd94e86ac64d38a63b", operator_name="Process:_Message", subtask_index="0", task_attempt_id="0ca06624eaeed4e7cb97441921b71a5b", task_attempt_num="0", task_id="e3dfc0d7e9ecd8a43f85f0b68ebf3b80", task_name="Source:_RECEIVER_A_OUT_____Process:_Call_____Map____Sink:__INT_MSG__Map____Sink:__ERROR___Process:_Message_____Map____Sink:_INT_MSG__Map____Sink:_ERROR__", tm_id="dbbde9b46d5423bbd046184c0c00f10d"}}]]
I0326 08:31:23.985434       1 resource_converter.go:170] unable to normalize group-resource operator_id from label "operator_id", skipping: no matches for {  operator_id}
I0326 08:31:23.985464       1 resource_converter.go:170] unable to normalize group-resource subtask_index from label "subtask_index", skipping: no matches for {  subtask_index}
I0326 08:31:23.985490       1 resource_converter.go:170] unable to normalize group-resource task_attempt_num from label "task_attempt_num", skipping: no matches for {  task_attempt_num}
I0326 08:31:23.985536       1 resource_converter.go:170] unable to normalize group-resource task_name from label "task_name", skipping: no matches for {  task_name}
I0326 08:31:23.985565       1 resource_converter.go:170] unable to normalize group-resource host from label "host", skipping: no matches for {  host}
I0326 08:31:23.985594       1 resource_converter.go:170] unable to normalize group-resource job_id from label "job_id", skipping: no matches for {  job_id}
I0326 08:31:23.985639       1 resource_converter.go:170] unable to normalize group-resource job_name from label "job_name", skipping: no matches for {  job_name}
I0326 08:31:23.985705       1 resource_converter.go:170] unable to normalize group-resource operator_name from label "operator_name", skipping: no matches for {  operator_name}
I0326 08:31:23.985753       1 resource_converter.go:170] unable to normalize group-resource task_attempt_id from label "task_attempt_id", skipping: no matches for {  task_attempt_id}
I0326 08:31:23.985794       1 resource_converter.go:170] unable to normalize group-resource task_id from label "task_id", skipping: no matches for {  task_id}
I0326 08:31:23.985833       1 resource_converter.go:170] unable to normalize group-resource instance from label "instance", skipping: no matches for {  instance}
I0326 08:31:23.985874       1 resource_converter.go:170] unable to normalize group-resource tm_id from label "tm_id", skipping: no matches for {  tm_id}
s-urbaniak commented 5 years ago

@avirammaryanchik most likely the problem is, that the adapter cannot map your metrics series to any kubernetes resources. According to your config you instructed the adapter to perform a 1:1 mapping to kubernetes resources:

      resources:
        template: "<<.Resource>>"

In the series there is only the job label which is mappable to the job resource: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#job-v1-batch which is probably not what you want.

I would suggest to inspect the kafka labels and see if it contains i.e. a pod label. That is most likely missing here.

ghost commented 5 years ago

Hi,

Will using an external metric server will work in this case?

s-urbaniak commented 5 years ago

yes, external metrics could help here if you can't map them to kubernetes resources.

RajalakshmiPrabhakar commented 5 years ago

Hi I am facing the same issue $kubectl logs custom-metrics-apiserver-58f7c8dbfb-np92v -n monitoring I0709 05:09:12.516848 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.516878 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.516899 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.516916 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.516937 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.516988 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.517010 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.517030 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.517048 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.517091 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.517112 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.517128 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.517147 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.517189 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.517210 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.517229 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance I0709 05:09:12.517260 1 resource_converter.go:188] unable to normalize group-resource endpoint from label "endpoint", skipping: no matches for /, Resource=endpoint I0709 05:09:12.517280 1 resource_converter.go:188] unable to normalize group-resource instance from label "instance", skipping: no matches for /, Resource=instance and unabel to get custom metrics