Cooldown logic for alert evaluation

To implement cooldown logic for alert evaluation, modify the WatchLogAlerts function in backend/jobs/log-alerts/log-alerts.go as follows:

Add a lastAlertTime map to track the last alert time for each alert:

lastAlertTime := make(map[int64]time.Time)

Update the processLogAlert function to include cooldown logic:

func processLogAlert(ctx context.Context, DB *gorm.DB, MailClient *sendgrid.Client, alert *model.LogAlert, rh *resthooks.Resthook, redis *redis.Client, ccClient *clickhouse.Client, lambdaClient *lambda.Client, lastAlertTime map[int64]time.Time) error {
    cooldownPeriod := time.Minute * 5 // Set cooldown period
    if lastTime, exists := lastAlertTime[alert.ID]; exists && time.Since(lastTime) < cooldownPeriod {
        log.WithContext(ctx).Infof("Alert %d is in cooldown period", alert.ID)
        return nil
    }

    // Existing logic...

    if alertCondition {
        lastAlertTime[alert.ID] = time.Now()
        // Existing alerting logic...
    }
    return nil
}

Pass lastAlertTime to processLogAlert calls in WatchLogAlerts:

err := processLogAlert(ctx, DB, MailClient, alert, rh, redis, ccClient, lambdaClient, lastAlertTime)

References

/backend/jobs/log-alerts/log-alerts.go /backend/jobs/log-alerts

#### About Greptile

This response provides a starting point for your research, not a precise solution. Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. [Ask Greptile](https://app.greptile.com/chat/github/highlight/highlight/main) · [Edit Issue Bot Settings](https://app.greptile.com/apps/github)

highlight / highlight

Cooldown logic for alert evaluation #8887

References