Investigate which CKF charms provide metrics by default

[orfeas-k]

Context

In order to evaluate better the work for integrating CKF charms with Observability charms, we need to investigate which CKF charms provide metrics by default. The state right now with charms alerts is documented in https://github.com/canonical/bundle-kubeflow/issues/837.

At the same time, we will document also the metrics that those charms provide.

What needs to get done

Document for every charm:

1. If it provides metrics by default
2. If yes, what are those metrics?

Definition of Done

We know for all CKF charms what they expose.

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5487.

This message was autogenerated

[orfeas-k]

admission-webhook

Code doesn't implement any metrics although there are some references in its go.mod and .sum files.

argo-controller

• Upstream docs on metrics exposed https://argo-workflows.readthedocs.io/en/latest/metrics
• argo-controller.txt

dex

• Code in upstream implementing metrics https://github.com/dexidp/dex/blob/088339fc287a24de23ffcfe7985287cef2a3b2fa/cmd/dex/serve.go#L26-L30
• dex-auth.txt

envoy

• Metrics include (they call them statistics) https://www.envoyproxy.io/docs/envoy/v1.15.0/configuration/upstream/cluster_manager/cluster_stats https://www.envoyproxy.io/docs/envoy/v1.15.0/configuration/http/http_conn_man/stats
• envoy.txt

istio-gateway

• charm endpoint: https://github.com/canonical/istio-operators/blob/e0d2c0261a3760366b87e7be27100393009967ac/charms/istio-gateway/src/manifest.yaml#L39-L40
• Upstream docs on metrics (it's an envoy): https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/statistics.html https://istio.io/latest/docs/ops/configuration/telemetry/envoy-stats/
• istio-gateway.txt

istio-pilot

• upstream docs https://istio.io/latest/docs/concepts/observability/
• istio-pilot.txt

jupyter-controller

• Provides default golang/prometheus and custom metrics
• jupyter-controller.txt

jupyter-ui

Code doesn't implement any metrics.

katib-controller

• Provides default golang/prometheus and custom metrics.
• katib-controller.txt

katib-db-manager

Code doesn't implement any prometheus metrics.

katib-ui

Code doesn't implement any metrics.

kfp-api

• Upstream endpoint https://github.com/kubeflow/pipelines/blob/5399585b6a0f92446bcfc5a7588f2a85ea0fe6a3/backend/src/apiserver/main.go#L208
• custom metrics in code
  • resource-manager
  • experiment-server
  • job-server
  • pipeline-server
  • pipeline-upload
  • run-server
• kfp-api.txt

Pipeline steps don't expose metrics by default. Feature requests:

• https://github.com/kubeflow/pipelines/issues/7341
• https://github.com/kubeflow/pipelines/issues/6336

kfp-metadata-writer

Code doesn't implement anything related to metrics https://github.com/kubeflow/pipelines/tree/master/backend/metadata_writer

kfp-persistence

Code doesn't implement any metrics. The only reference to metrics is about "metrics" provided from the application for exposing artifacts in the UI.

kfp-profile-controller

Code doesn't implement any metrics.

kfp-schedwf

Code doesn't implement any prometheus metrics.

kfp-ui

Code doesn't implement any prometheus metrics.

kfp-viewer

Code doesn't implement any prometheus metrics.

kfp-viz

Code doesn't implement any prometheus metrics.

knative-eventing & knative-serving

• They do expose metrics
  • knative-serving metrics
  • knative-eventing metrics
• However, through one shared exporter (can't use for querying up metric of those charms)
• knative-eventing-serving.txt

knative-operator

• Exposes metrics through this endpoint (but charm doesn't configure its service right now)
• knative-operator.txt

kserve-controller

• Workload exposes metrics at endpoint (uses /metrics by default) although our charm doesn't patch the service to expose this endpoint.
• Implements some kind of default k8s metrics.
• kserve-controller.txt Side note that if we 'd like ISVCs to expose metrics too (via their own endpoints - docs), we should modify the values in the configmap according to this guide.

kubeflow-dashboard

• implements metrics but are only accessible in the case of a deployment in GCE. When deploying CKF, that's what the dashboard logs:

  2024-04-02T08:03:21.120Z [serve] > kubeflow-centraldashboard@0.0.2 serve
  2024-04-02T08:03:21.120Z [serve] > node dist/server.js
  2024-04-02T08:03:21.120Z [serve] 
  2024-04-02T08:03:23.555Z [serve] Initializing Kubernetes configuration
  2024-04-02T08:03:23.611Z [serve] Unable to fetch Application information: 404 page not found
  2024-04-02T08:03:23.611Z [serve] 
  2024-04-02T08:03:23.637Z [serve] "other" is not a supported platform for Metrics
  2024-04-02T08:03:23.638Z [serve] Using Profiles service at http://kubeflow-profiles.kubeflow:8081/kfam
  2024-04-02T08:03:23.645Z [serve] Server listening on port http://localhost:8082 (in production mode)

• Curling the pod's endpoint <pod-ip>:80802/api/metrics returns

  {"error":"Operation not supported"}%

  Thus, metrics are not available from upstream in our case.

kubeflow-profiles

kfam

• implements metrics at endpoint
• golang/prometheus and custom metrics
• kfam.txt

profiles

• implements metrics at endpoint
• golang/prometheus and custom metrics
• kubeflow-profiles.txt

kubeflow-roles

There isn't an upstream app for this charm.

kubeflow-volumes

Code doesn't implement any metrics.

metacontroller

• Implements golang/prometheus plus some custom metrics at endpoint (already exposed by charm)
• metacontroller-operator.txt

minio

• Upstream docs on metrics exposed at endpoints with some info
• minio.txt

mlmd

Code doesn't implement any metrics.

oidc-gatekeeper

There are some references to prometheus packages in go.mod and .sum files but nothing is implemented in its code.

pvcviewer-operator

• Code defines a metric endpoint and also has the corresponding manifests too (auth endpoint, serviceMonitor)
• However, I couldn't access those although the charm exposes the port

  k exec envoy-operator-0 -n kubeflow -- curl pvcviewer-operator.kubeflow.svc.cluster.local:8443/metrics
  Client sent an HTTP request to an HTTPS server.

  This is probably due to this note in upstream manifests. We should probably remove this from our charm command (although I didn't try and need to verify this). It could also be that we have not included the Prometheus manifests in our case.

• Thus, no text file containing exposed metrics here.

seldon-controller-manager

• Implements metrics.
• Upstream docs on metrics and here
• seldon-core.txt

tensorboard-controller

• Implements metrics and defines an endpoint in manifests. This also (like pvcviewer-operator) has references about putting metrics behind auth so we should be careful when implementing it (looks like now it's not behind auth)
• tensorboard-controller.txt

tensorboard-web-app

Code doesn't implement any metrics.

training-operator

• Metrics are exposed in endpoint
• Golang/prometheus and custom metrics are implemented. Custom metrics are also described here.
• (not 100% releveant but) Jobs implement some custom metrics too (references 1, 2 and 3)
• training-operator.txt

[orfeas-k]

Upstream apps that do not already expose metrics (wip)

1. admission-webhook
2. jupyter-ui (jupyter-web-app)
3. katib-db-manager
4. katib-ui
5. kfp-metadata-writer
6. kfp-persistence
7. kfp-profile-controller
8. kfp-schedwf
9. kfp-ui
10. kfp-viewer
11. kfp-viz
12. kubeflow-dashboard (explanation in previous comment)
13. kubeflow-roles. This isn't an upstream app but we 'd still need an exporter if we 'd like metrics from this charm.
14. kubeflow-volumes (volumes-web-app)
15. mlmd
16. oidc-authservice
17. tensorboards-web-app

[kimwnasptd]

Regarding all the K8s Controllers from kubeflow/kubeflow (notebooks, profiles, tensorboards) they will get some quite useful metrics by default because of controller-runtime golang package, that comes with Kubebuilder https://book.kubebuilder.io/reference/metrics-reference

Those are perfect for capturing if the controllers are working as expected, and it's great it will be handled by default.

In order for this to happen though, someone upstream will need to bump the controller-runtime package from 0.11 to 0.16.3

[orfeas-k]

Related sheet