Traction database instances being heavily throttled in Traction (bc0192) environments

[WadeBarnes]

The Traction database instances are being throttled at >60% on average. Review and adjust the CPU resource allocations, primarily the CPU limit to reduce or eliminate the throttling. The goal should be to reduce throttling to <25% on average. For production an even lower average may be desirable.

These metrics can be easily reviewed using the Namespace Monitoring dashboard available through Grafana in our new monitoring stack.

[WadeBarnes]

bc0192-dev

• CPU limit for exporter container in traction-database-ha-* stateful sets should be increased from 50m to 250m
• CPU limit for pgbackrest container in traction-database-ha-* stateful sets should be increased from 250m to 500m

bc0192-test

• CPU limit for exporter container in traction-database-ha-* stateful sets should be increased from 50m to 250m
• CPU limit for pgbackrest container in traction-database-ha-* stateful sets should be increased from 250m to 500m

bc0192-prod

• CPU limit for exporter container in traction-database-ha-* stateful sets should be increased from 50m to 250m
• CPU limit for pgbackrest container in traction-database-ha-* stateful sets should be increased from 250m to 500m

[WadeBarnes]

@i5okie, Where should these changes (values file) be made?

[i5okie]

These get applied directly to postgrescluster CR. these live in trust-over-ip-configurations repo. Applied with kubectl apply -k bc0192/dev From the kustomize directory.

Though, a couple have been applied with ArgoCD. In ministry-gitops-ditp repo.

[i5okie]

• postgres-cluster base resource has been updated to reflect these changes, and pushed to the ministry-gitops-ditp repo.
• Changes have been applied to bc0192- dev, test, and prod via ArgoCD

Will continue monitoring in Grafana to make sure the changes have fixed the issue.

[i5okie]

Changes successfully resolved the throttling issue.