Recently the BGS service went down and this helped us understand that the BGS container lost connection to the rabbitMQ broker. This left the service unusable for partner team applications. We were not notified that this happened by any custom VRO monitoring. Therefore we need to ensure that in the future we can detect when this issue happens and are alerted.
AC
Implement monitoring and alerting for a similar issue like this
There should be a monitor specifically for the RabbitMQ connection
Notes
Lisa and Erik both contributed to the BGS monitoring developed in #2980. That work is scheduled to be released during Sprint Y. While these metrics would have helped us identify the BGS outage, Mason wisely advised that we should have monitoring that triggers a clear pass/fail condition so that RabbitMQ connection issues would immediately notify the team for escalation into our incident response plan. Feel free to contact Lisa or Erik for an overview of the metrics implemented in #2980. That prior work may well be relevant to the monitoring needed here.
What and why:
Recently the BGS service went down and this helped us understand that the BGS container lost connection to the rabbitMQ broker. This left the service unusable for partner team applications. We were not notified that this happened by any custom VRO monitoring. Therefore we need to ensure that in the future we can detect when this issue happens and are alerted.
AC Implement monitoring and alerting for a similar issue like this There should be a monitor specifically for the RabbitMQ connection
Notes
Datadog log from the incident https://lighthousedi.ddog-gov.com/account/login?next=%2Fdashboard%2Fpvw-4a2-2s5%2Fvro-on-lhdi%3FfromUser%3Dfalse%26fullscreen_end_ts%3D1719892740000%26fullscreen_paused%3Dtrue%26fullscreen_refresh_mode%3Dpaused%26fullscreen_section%3Dedit%26fullscreen_start_ts%3D1719806400000%26fullscreen_widget%3D19%26refresh_mode%3Dsliding%26tpl_var_cluster%255B0%255D%3Dldx-prod-1%26tpl_var_deployment%255B0%255D%3Dvro-svc-bgs-api%26tpl_var_namespace%255B0%255D%3Dva-abd-rrd-prod%26view%3Dspans%26from_ts%3D1719935942514%26to_ts%3D1719939542514%26live%3Dtrue