SysDig Alerts for DB - Githubissues

bcgov / SIMS

Student Information Management System. Post-Secondary Student Financial Aid System

Apache License 2.0

25 stars 14 forks source link

SysDig Alerts for DB #3906

Closed ninosamson closed 2 days ago

ninosamson commented 2 weeks ago

Need to re-configure SysDig alerts for database as they're still pointing to patroni.

Acceptance Criteria

Consolidated Acceptance Criteria for Sysdig Alerts

PVC Storage Utilization

[x] Trigger: PVC storage utilization exceeds 80%.
- Environments: DEV, TEST, STG, PROD.
- Severity: Medium for DEV/TEST/STG, High for PROD.

Container CPU and Memory Usage

[x] CPU Throttling
- Trigger: CPU usage exceeds 90% in STG or PROD.
[x] Memory Usage
- Trigger: Memory usage exceeds 90% in STG or PROD.

Init Container and Pod Issues

[x] Init Container Terminated with Error
- Trigger: An init container in STG or PROD terminates with an error.
[x] Pod Container Creating Timeout
- Trigger: A pod in STG or PROD remains in "ContainerCreating" state for over 10 minutes.
[x] Pod Container Terminated with Error
- Trigger: A pod in STG or PROD terminates with an error.

Job Completion Failure

[x] Trigger: Any job in PROD fails to complete.

StatefulSet Pod Readiness

[x] Backup StatefulSet Pod Not Ready
- Trigger: Crunchy DB Backup StatefulSet pods in STG or PROD are not ready.
[x] Primary StatefulSet Pods Not Ready
- Trigger: StatefulSet pods (CrunchyDB, MongoDB, Redis) in STG or PROD are not ready.

Notification Rules

[x] Persistence:
- Medium Severity: Teams notifications.
- High Severity: Teams notifications.

andrewsignori-aot commented 2 weeks ago

@guru-aot to add ACs as required.