bcgov / DITP-DevOps

Digital Identity and Trust Program Team's DevOps Documentation Repository
Apache License 2.0
2 stars 5 forks source link

Update OpenShift resources in Traction Test to the “load test” levels in Sandbox and revert Sandbox to pre-Load Test levels #153

Closed swcurran closed 5 months ago

swcurran commented 6 months ago

We will be using Test for load testing vs. Sandbox. Please update the resources in Test to match what we raised Sandbox to when we started load testing, and revert Sandbox to the pre-load testing levels.

WadeBarnes commented 6 months ago

Aaca-py instances have been configured with:

  - resources:
      limits:
        cpu: '1'
        memory: 1600Mi
      requests:
        cpu: 200m
        memory: 820Mi
i5okie commented 6 months ago

CrunchyDB PostgreSQL Cluster has been updated with:

resources:
  limits:
    cpu: '2'
    memory: 4000Mi
  requests:
    cpu: 200m
    memory: 820Mi
WadeBarnes commented 6 months ago

Over to @swcurran for testing.

swcurran commented 6 months ago

Looking good! Much better than before. Thanks! I just completed a 10 minute test — anything interesting on the load/throttling/queries level?

WadeBarnes commented 6 months ago

A bit of throttling on the database instances, but it's hard to tell if that was affected by the load test, since it was already at that level (i.e. load test made no change). image

WadeBarnes commented 6 months ago

@i5okie, do we have enough CPU allocation to up the DB CPU limit to 3 or 4? Could we move things around temporarily to accommodate for a load test? The DB throttling is still a bit higher than I'd like. I'd like to see whether more CPU will help.

i5okie commented 6 months ago

Limit updated to 3 CPU. There is no more room left for another upgrade.

swcurran commented 6 months ago

The big issue is how to find out what queries are running on the DB. Is that available? Its likely that there is a query or two that is hurting performance.

swcurran commented 5 months ago

I think this is done. Closing — let me know if I’m not right.