[BUG] Not all Istio component metrics are available via Prometheus

heycait commented 4 years ago

Summary

A Prometheus server deployed to cf-system can't scrape all Istio component metrics.

Using the following NetworkPolicy:

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
  namespace: istio-system
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          cf-for-k8s.cloudfoundry.org/cf-system-ns: ""
      podSelector:
        matchLabels:
          what-am-i: prometheus

And adding the following annotations to all the Istio deployments (istio-citadel, istio-galley, istio-pilot, etc):

prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "15014"

Only metrics for citadel, pilot, and sidecar-injector are exposed. Metrics such as:

citadel_secret_controller_svc_acc_created_cert_count{app="citadel",instance="10.44.1.26:15014",istio="citadel",job="kubernetes-pods",kapp_k14s_io_app="1594227271826566527",kapp_k14s_io_association="v1.98d4885da04b76e7e3e6f3bcef9e11f1",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-citadel-d696cdcd4-8zpv2",pod_template_hash="d696cdcd4"}
istio_mcp_message_sizes_bytes_bucket{app="pilot",chart="pilot",collection="istio/authentication/v1alpha1/meshpolicies",component="pilot",heritage="Tiller",instance="10.44.5.22:15014",istio="pilot",job="kubernetes-pods",kapp_k14s_io_app="1594227271826566527",kapp_k14s_io_association="v1.46f9fdb3ee40911a0a50605f1d48cba8",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-pilot-8c74c5b74-588vc",le="420230.4",pod_template_hash="8c74c5b74",release="istio"}
sidecar only seems to provide Go metrics and nothing particular about the sidecar go_gc_duration_seconds{app="sidecarInjectorWebhook",chart="sidecarInjectorWebhook",heritage="Tiller",instance="10.44.3.33:15014",istio="sidecar-injector",job="kubernetes-pods",kapp_k14s_io_app="1594227271826566527",kapp_k14s_io_association="v1.b726a0d7664b8c79ac1b06e7504fc13d",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-sidecar-injector-67fbb95599-qgv2x",pod_template_hash="67fbb95599",quantile="0",release="istio"}

Metrics for galley, telemetry, and policy do not work

Deployment Configuration

cf-for-k8s version: https://github.com/cloudfoundry/cf-for-k8s/commit/9db38f77ff7d290573a90a29c7421bb7844fe83d
cf-k8s-networking version: ref: v0.0.6
Deploy command: Used the exact deploy steps mentioned in cf-for-k8s, no extra configuration

Kubernetes CLI and API version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:34:25Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.10-gke.8", GitCommit:"fd29f95f189cb16dd8fe0bd5dc8b485dfd049d2c", GitTreeState:"clean", BuildDate:"2020-06-23T23:44:30Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

IaaS: GKE

[Describe any other special configuration here]

Prometheus was deployed as described here: https://github.com/cloudfoundry/cf-for-k8s-metric-examples#deploying-prometheus-server

Reproduction Steps

What steps/actions led to the issue? Wanted to check if istio component metrics were scrapeable via Prometheus UI if I added the appropriate annotations.

Logs

It's helpful to include snippets of the error response or logs output The failing components (galley, telemetry, and policy) show an error of server returned HTTP status 503 Service Unavailable when Prometheus tries to scrape it.

Expected behavior

up{kubernetes_namespace="istio-system"} query in Prometheus UI shows a successfully scrape of 1 for all istio components.

Additional context

I based the Prometheus annotations off of these files which mention the port 15014:

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173757860

The labels on this github issue will be updated when the story is started.

jenspinney commented 4 years ago

Thanks for reporting this @heycait! We'll need to explore a little to understand why metrics aren't working on those particular components.

KauzClay commented 4 years ago

Hey @heycait ,

We are in the process of upgrading to istio 1.6.x, which no longer has separate pods for galley, telemetry, and policy. It just has the single istiod pod.

We still need to validate metrics from this new pod, but in the meantime, we wouldn't worry about getting metrics from galley, telemetry, or policy.

kauana commented 4 years ago

Hello @heycait :wave: ,

We upgraded istio to 1.6.4 and went through your reproduction steps and did not see any metrics for galley, telemetry and policy. This is because these components were all merged into a single component called istiod.

Here is a picture of what up{kubernetes_namespace="istio-system"} looks like in Istio 1.6.4: prometheus

cloudfoundry / cf-k8s-networking