We had a case where quite some things were happening and a lot of service were trying to send alerts.
The task queue of the alerta service was extremely long and make the service constantly sending request to the alerta server.
Result of this was all the bandwith and CPU of the robot was used to send those alerts and the robot was very hard to talk to because of that.
We should find a way to throttle the actions of the service and prevent this from happening again.
We had a case where quite some things were happening and a lot of service were trying to send alerts. The task queue of the alerta service was extremely long and make the service constantly sending request to the alerta server. Result of this was all the bandwith and CPU of the robot was used to send those alerts and the robot was very hard to talk to because of that.
We should find a way to throttle the actions of the service and prevent this from happening again.