Load Watcher

The load watcher is responsible for the cluster-wide aggregation of resource usage metrics like CPU, memory, network, and IO stats over time windows from a metrics provider like SignalFx, Prometheus, Kubernetes Metrics Server etc. developed for Trimaran: Real Load Aware Scheduling in Kubernetes. It stores the metrics in its local cache, which can be queried from scheduler plugins.

The following metrics provider clients are currently supported:

1) SignalFx 2) Kubernetes Metrics Server 3) Prometheus

These clients fetch CPU usage currently, support for other resources will be added later as needed.

Tutorial

This tutorial will guide you to build load watcher Docker image, which can be deployed to work with Trimaran scheduler plugins.

The default main.go is configured to watch Kubernetes Metrics Server. You can change this to any available metrics provider in pkg/metricsprovider. To build a client for new metrics provider, you will need to implement FetcherClient interface.

From the root folder, run the following commands to build docker image of load watcher, tag it and push to your docker repository:

docker build -t load-watcher:<version> .
docker tag load-watcher:<version> <your-docker-repo>:<version>
docker push <your-docker-repo>

Note that load watcher runs on default port 2020. Once deployed, you can use the following API to read watcher metrics:

GET /watcher

This will return metrics for all nodes. A query parameter to filter by host can be added with host.

Metrics Provider Configuration

• By default Kubernetes Metrics Server client is configured. Set KUBE_CONFIG env var to your kubernetes client configuration file path if running out of cluster.

• To use the Prometheus client, please configure environment variables METRICS_PROVIDER_NAME, METRICS_PROVIDER_ADDRESS and METRICS_PROVIDER_TOKEN to Prometheus, Prometheus address and auth token. Please do not set METRICS_PROVIDER_TOKEN if no authentication is needed to access the Prometheus APIs. Default value of address set is http://prometheus-k8s:9090 for Prometheus client.

• To use the SignalFx client, please configure environment variables METRICS_PROVIDER_NAME, METRICS_PROVIDER_ADDRESS and METRICS_PROVIDER_TOKEN to SignalFx, SignalFx address and auth token respectively. Default value of address set is https://api.signalfx.com for SignalFx client.

Deploy load-watcher as a service

To deploy load-watcher as a monitoring service in your Kubernetes cluster, you should replace the values in the [] with your own cluster monitoring stack and then you can run the following.

> kubectl create -f manifests/load-watcher-deployment.yaml

Using load-watcher client

• load-watcher-client.go shows an example to use load-watcher packages as libraries in a client mode. When load-watcher is running as a service exposing an endpoint in a cluster, a client, such as Trimaran plugins, can use its libraries to create a client getting the latest metrics.