search

bcgov / DITP-DevOps

Digital Identity and Trust Program Team's DevOps Documentation Repository
Apache License 2.0
2 stars 5 forks source link

Review production agent configuration #152

Closed WadeBarnes closed 5 months ago

WadeBarnes commented 6 months ago

IDIM agent was running a single instance in prod, when we've had it configured to run three instances. That instance was interrupted by platform updates, causing a 6 minute outage for the service.

Review the configuration of the production services to ensure they are HA.

WadeBarnes commented 6 months ago

During the review I found the old vc-authn instances were still running in dev, test, and prod. After talking with @esune I've scaled these down to see if anyone complains. All services should be using the new instances now.

WadeBarnes commented 6 months ago

The configuration of the following agents has been updated to include HPAs and anti-affinity rules to ensure a minimum number of pods are running and they are explicitly distributed across compute nodes.

WadeBarnes commented 6 months ago

@esune, Do you know the current operational status of the Buy BC issuers?

WadeBarnes commented 6 months ago

@esune, Do you also know the current operational status of the LCRB issuers? They already have HPA configurations but are all set to min 1 max 1 and they are only operational in dev and test, the prod instance is scaled to 0.

WadeBarnes commented 6 months ago

Also reviewed the mediator and endorser instance configurations, they both have HPA configurations and are scaled appropriately.

WadeBarnes commented 6 months ago

BC Wallet Demo has HPA configurations and is scaled appropriately.

WadeBarnes commented 6 months ago

Review of the following environments is complete (pending any feedback):

esune commented 5 months ago

@WadeBarnes I am not sure on whether the LCRB and BuyBC issuer agents are currently being used - I suspect they only issue credentials sporadically, if any.

WadeBarnes commented 5 months ago

@esune, What's the best way to find out? I don't have contacts for either.

esune commented 5 months ago

@esune, What's the best way to find out? I don't have contacts for either.

I'll try to track things down

esune commented 5 months ago

Both LCRB and BuyBC agents seem to not be in use right now. We will spin them down and see if we receive inquiries/complaints and act accordingly.

@WadeBarnes if these were the last two agents requiring review we might be able to close this issue.