[Observability] Core and Custom Metrics in Radius

[ytimocin]

Core and Custom Metrics in Radius

Author: Yetkin Timocin (@ytimocin) Last Updated: 11/11/2022 Status: [Work in Progress]

This document proposes a design for the set of metrics in Radius.

• Introduction
• Core Metrics in Radius
  • Details
• Custom Metrics in Radius
  • Introduction
  • Design
  • Proposed Custom Metrics
• Future Work
• References

Introduction

Core Metrics in Radius

(https://margara.faculty.polimi.it/papers/2020_debs_kaiju.pdf)

Details

Custom Metrics in Radius

Introduction

Design

Proposed Custom Metrics

[Draft]

Radius Core Metrics

• radius/core/async_queue_message_count: Number of messages in the queue

Core RP Metrics

• radius/appcorerp/corerp/server_request_count: Number of requests started in server
• radius/appcorerp/corerp/server_request_bytes: Request body size if set as ContentLength (uncompressed) in server
• radius/appcorerp/corerp/server_response_bytes: Response body size (uncompressed) in server
• radius/appcorerp/corerp/server_latency: Request end to end latency in server

Link RP Metrics

• radius/appcorerp/linkrp/server_request_count: Number of requests started in server
• radius/appcorerp/linkrp/server_request_bytes: Request body size if set as ContentLength (uncompressed) in server
• radius/appcorerp/linkrp/server_response_bytes: Response body size (uncompressed) in server
• radius/appcorerp/linkrp/server_latency: Request end to end latency in server

CLI Metrics

• radius/cli/sent_bytes: Total bytes sent in request body (not including headers)
• radius/cli/received_bytes: Total bytes received in response bodies (not including headers but including error responses with bodies)
• radius/cli/roundtrip_latency: End-to-end latency
• radius/cli/completed_count: Count of completed requests

Deployment Engine Metrics

• radius/de/sent_bytes: Total bytes sent in request body (not including headers)
• radius/de/received_bytes: Total bytes received in response bodies (not including headers but including error responses with bodies)
• radius/de/roundtrip_latency: End-to-end latency
• radius/de/completed_count: Count of completed requests

References

1. https://margara.faculty.polimi.it/papers/2020_debs_kaiju.pdf
2. https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/monitoring_architecture.md
3. https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md
4. https://opentelemetry.io/docs/

AB#5058

[youngbupark]

• Why do we need to collect CLI metrics? How to collect user's data? will we spin off ingestion pipeline for this? As we spoke, CLI is out of scope in the current metrics work since CLI is running on user context, not radius service context. Also we can collect CLI behaviors from server side.
• Can you define DE metrics consistent with the other RP? these are server side metrics so it should be same name and dimentions.
• What are the goal and non-goal of this proposal?
• Gathering Kubernetes metrics is out of scope. This needs to focus only on radius metrics
• What is the proposal of telemetry backend service infra? which telemetry client will you ? e.g. opentelemetry
• The below metrics are missing:
  1. Client sdk metrics too such as kubeclient/azure client sdk/UCP client/etc.
  2. Backend worker metrics such as how many operation is failed or succeeded.
• What kinds of minimum attributions(a.k.a dimensions) will be included in each metrics?

[ytimocin]

Created a gist (https://gist.github.com/ytimocin/fb16dc08f3e7352ef7f0740b2708d58c) and closing this. Can be followed through the original issue: https://github.com/project-radius/radius/issues/4262.