storj / team-metainfo

GNU Affero General Public License v3.0
0 stars 0 forks source link

Create alerts for accounting by buckets execution time #102

Closed mniewrzal closed 2 years ago

mniewrzal commented 2 years ago

We have alerts about too long segments loop execution time for all satellites but we don't have similar thing for accounting loop that is iterating over objects. We should monitor service.Tally function execution time from tally package. We need alerts for all production satellites. Initial execution time to trigger alert should be 2h. Alerts need to send notifications to slack channel #team-infrastructure.

mniewrzal commented 2 years ago

https://redash.datasci.storj.io/queries/2772?p_GROUP_BY=1h#4429 general dashboard https://redash.datasci.storj.io/queries/2773#4431 us1 https://redash.datasci.storj.io/queries/2775#4435 eu1 https://redash.datasci.storj.io/queries/2774#4433 ap1