search

linkerd / linkerd2

Ultralight, security-first service mesh for Kubernetes. Main repo for Linkerd 2.x.
https://linkerd.io
Apache License 2.0
10.65k stars 1.28k forks source link

provide custom metrics APIService #2822

Open siggy opened 5 years ago

siggy commented 5 years ago

Summary

Kubernetes provides a Horizontal Pod Autoscaler to enable scaling pods based on metrics. These metrics may come from the container (cpu, mem), or, via a custom metrics APIService.

The autoscaling thanos-demo demonstrates using a k8s-prometheus-adapter to read Linkerd latency metrics, expose them via a custom metrics APIService, and affect scaling via an HPA.

Proposal

Implement a custom metrics API service in Linkerd's control-plane. This would enable autoscaling out-of-the-box, and also enable Linkerd's metrics to be easily consumable via systems that expect Kubernetes metrics.

grampelberg commented 5 years ago

See https://github.com/grampelberg/leaderboard for how to get this working!

kevingbb commented 5 years ago

It would be great to be able to leverage linkerd custom metrics such as rps, success rate and latency without having to install a custom metrics adapter like k9s-prometheus-adapter for example.

The additional challenge I foresee in this is the fact that there can only be a single custom metrics adapter per k8s installation. So if the prometheus adapter has been installed already, or something like keda has already been installed, then a linkerd custom metrics adapter would overwrite both. Or maybe my misunderstanding of the custom and external metrics endpoints in k8s is wrong.

grampelberg commented 5 years ago

then a linkerd custom metrics adapter would overwrite both

You're correct, there can only be one!

geekyfox90 commented 4 years ago

The solution is to use Prometheus federation API in order to copy linkerd metrics to the prometheus linked with your prometheus adapter.

Check out this article on how to use the federation API to get linkerd metrics on your prometheus used by the prometheus adapter. https://linkerd.io/2/tasks/exporting-metrics/

I tested this and it works perfectly :)

naseemkullah commented 4 years ago

The solution is to use Prometheus federation API in order to copy linkerd metrics to the prometheus linked with your prometheus adapter.

Any downside to connecting the prom adapter to the copied metrics prometheus rather than at the source linkerd-prometheus (e.g. less acurate, less precise, less reactive or anything of the sort)?

grampelberg commented 4 years ago

@naseemkullah there'll be a delay and it will be dependent on how you've configured your own prometheus installation.