The incident started at 2024-02-21 02:07 GMT and was fully resolved at 2024-02-21 05:34 GMT. This is a manual addition because it wasn't detected by automated downtime detection tools.
The issue was the connection between the Config Broker and our secret manager which caused the Config Broker to be unable to refresh its database access credentials before the credentials expired.
Compounding the issue, our monitoring continued to show Config Broker as being healthy and showing the app with normal access to the database. This turned out to be a separate issue with our monitoring logic. This has now been remedied so that any similar issues in the future will alert us right away.
While Config Broker was degraded, no configuration changes were possible by vendors or tenants. However, the issue did not impact any running Tenant Security Proxies, which means that there would not have been any service impact to customers aside from being unable to change their configuration during the incident.
The incident started at 2024-02-21 02:07 GMT and was fully resolved at 2024-02-21 05:34 GMT. This is a manual addition because it wasn't detected by automated downtime detection tools.
The issue was the connection between the Config Broker and our secret manager which caused the Config Broker to be unable to refresh its database access credentials before the credentials expired.
Compounding the issue, our monitoring continued to show Config Broker as being healthy and showing the app with normal access to the database. This turned out to be a separate issue with our monitoring logic. This has now been remedied so that any similar issues in the future will alert us right away.
While Config Broker was degraded, no configuration changes were possible by vendors or tenants. However, the issue did not impact any running Tenant Security Proxies, which means that there would not have been any service impact to customers aside from being unable to change their configuration during the incident.