department-of-veterans-affairs / abd-vro

To get Veterans benefits in minutes, VRO software uses health evidence data to help fast track disability claims.
Other
19 stars 6 forks source link

Alerting rabbitmq metrics in datadog #3482

Open lisac opened 1 month ago

lisac commented 1 month ago

[!IMPORTANT] respond by 9/24 to https://github.com/department-of-veterans-affairs/lighthouse-di-tenant-support/issues/39 (add a comment to the ticket) on whether we will pursue next steps; and if so, which engineer will work with Tyler

User Story

As a VRO engineer, I would like the ability to view rabbitmq metrics in datadog at the pod level (per environment), so that I can better troubleshoot issues and refine failover policies. At this time, we are only able to monitor Rabbitmq connection on a global scale per VRO applications but we can't pinpoint which environment(s) is specifically affected when there is a drop in Rabbitmq connections.

Acceptance Criteria

Not included in this work Here are tickets that handle monitoring of apps (BIP, BGS) individually and they also monitor their connectivity to RabbitMQ: #3017 #3018

lisac commented 1 month ago

hi @Ponnia-M , i noticed Tyler asked for confirmation/feedback on what appears to be a metrics dashboard - his comment in https://github.com/department-of-veterans-affairs/lighthouse-di-tenant-support/issues/39#issuecomment-2375302908 - is that something you can respond to?

Ponnia-M commented 3 weeks ago

I created a new LHDI issue in regards to this