Sentry Percentage consumed increses for past few days. - Githubissues

ministryofjustice / operations-engineering

This repository is home to the Operations Engineering's tools and utilities for managing, monitoring, and optimising software development processes at the Ministry of Justice. • This repository is defined and managed in Terraform

https://user-guide.operations-engineering.service.justice.gov.uk/

MIT License

14 stars 5 forks source link

Sentry Percentage consumed increses for past few days. #4513

Closed vijaykannan21 closed 5 months ago

vijaykannan21 commented 5 months ago

We have been receiving the Sentry Alert for the past three days for overconsumption.

Sentry Transaction Usage Alert :

Usage threshold: 80%
Period: 1 day
Max usage for period: 967741 Transactions
Transactions consumed over period: 957286
Percentage consumed: 99%

vijaykannan21 commented 5 months ago

The list of projects consumed nearly 100 K sentry transaction rates in the past three days.

Project

Workforce-allocations-to-delicus. Team - Probation integration
hmpps-approved-premises-api. Team - hmpps-approved-permises , hmpps-cas-2, hmps-temporary-accommodation
laa-crime-application-store-dev. Team - laa-claim-non-standard-magistrate
domain-events-and-delius Team - probation- integration
Find-moj-data Team - data-catelogue-devs
family-mediators-api-prod Team - central-digital-product-team
hmpps-accredited-programmes-api. Team - hmpps-accredited-programmes
laa-crime-application-store-prod. Team- laa-claim-non-standard-magistrate
laa-crime-application-store-uat. Team - laa-claim-non-standard-magistrate

vijaykannan21 commented 5 months ago

Contacting the specific teams to collect the reason for sudden increase in transaction rate.

vijaykannan21 commented 5 months ago

Update from project find-moj-data Team - alerts that were generated and it seems that a large number were from incorrect environment settings in local development. We have agreed that we will add a flag to ignore sentry in local development which should stop future reoccurrence.

vijaykannan21 commented 5 months ago

Just communicated with the project team and check the transaction rate and recommended them to use the daily quota and transaction rate with in the daily limit.