metrics: add count for number of upscale and downscale

fl0Lec commented 3 months ago

Motivation

Currently the wpa controlleur is only exposing a logs to inform of any upscale/downscale. It would be very useful to have a metric on upscale/downscale in order to be able to visualize them.

see: https://github.com/DataDog/watermarkpodautoscaler/blob/9adcbcf5a9df4bb6b4370820dead417afa7c6932/controllers/watermarkpodautoscaler_controller.go#L379

Scope

Create two count metrics:

watermarkpodautoscaler.wpa_controller_upscale_replicas_total: count the number of upscale done
watermarkpodautoscaler.wpa_controller_downscale_replicas_total: count the number of down scale

What does this PR do?

A brief description of the change being made with this pull request.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Describe your test plan

Write there any instructions and details you may have to test your PR.

fl0Lec commented 3 months ago

I do not have the right to add the label to this PR but this is an enhancement

fl0Lec commented 3 months ago

This can be obtain using the watermarkpodautoscaler.wpa_controller_replicas_scaling_effective with a diff function

clamoriniere commented 3 months ago

After some discussion with @fl0Lec

we think that the new metrics can be useful to keep track on every scale actions made by the WPA controller. Controller already provide Kubernetes events when it trigger a scale action on a Deployment/Statefulset... but it is not easy to extract the number of replicas the controller is adding or removing.

DataDog / watermarkpodautoscaler