search

docker-flow / docker-flow-monitor

MIT License
87 stars 38 forks source link

scaling to 0 removes alert rules #35

Closed patrickleet closed 6 years ago

patrickleet commented 6 years ago

I am trying to get the benefits of serverless, with microservices by autoscaling based on queue length. When there are no messages in the queue, the service can be scaled down to 0.

I'm using @thomasjpfan's https://github.com/thomasjpfan/docker-scaler, though Jenkins job would be the same, and a rabbitmq prometheus exporter. This allows me to scale without instrumenting the service specifically.

~ $ docker service logs voelhgexsur4
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal    | 2018/02/20 07:21:32 Scale service down: rethink-denormalizer_service
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal    | 2018/02/20 07:21:32 scale-service success: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal    | 2018/02/20 07:21:32 Alertmanager received message: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal    | 2018/02/20 07:31:32 Scale service down: rethink-denormalizer_service
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal    | 2018/02/20 07:31:32 scale-service success: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal    | 2018/02/20 07:31:32 Alertmanager received message: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)

It works great.

Problem is, this bring the service down in the eyes of DFM, and correctly, for DFP.

In DFM this is an issue because now, because the alerts needed to scale back up when there are messages in the queue again have been removed as part of the service removed event.

patrickleet commented 6 years ago

would it be possible to add a 

`com.df.alertPersistent.1: true`

label that would tell DFM to keep it even though the service was removed?

thomasjpfan commented 6 years ago

It is possible to add the com.df.alertPersistent label to persistent alerts when services are scaled to zero. If @vfarcic thinks this is a good feature, I can work on a PR for this.

vfarcic commented 6 years ago

I think it's a good feature :)

vfarcic commented 6 years ago

It's available in the release 18.02.21-52