search

defenseunicorns / uds-core

A FOSS secure runtime platform for mission-critical capabilities
https://uds.defenseunicorns.com
GNU Affero General Public License v3.0
52 stars 21 forks source link

HA Keycloak Fails to Gracefully Upgrade Major Versions #1009

Closed joelmccoy closed 1 week ago

joelmccoy commented 1 week ago

Environment

Device and OS: AWS rke2 1.29 App version: Keycloak 25 -> 26 Kubernetes distro being used: rke2 1.29 Other: uds-core v0.30 -> v0.31

Steps to reproduce

  1. Upgrade from uds-core v0.30 -> v0.31 with an HA Keycloak Deployment

Expected result

Graceful upgrade of keycloak and uds-core

Actual Result

1 of the 3 keycloak pods stuck in crashloop and kicking the pods did not resolve the issue

Visual Proof (screenshots, videos, text, etc)

Some snippets of noteable pod log errors:

ERROR: Failed to start server in (production) mode
ERROR: Unsupported protocol version 152

Severity/Priority

Probably high? Essentially anyone who rolls keycloak HA is going to run into this issue on this upgrade it seems

Additional Context

Apparently Keycloak doesn't really support rolling major upgrades according to this comment so we may need to find a workaround to support these upgrades.

joelmccoy commented 1 week ago

Additional context: this happened in two different environments (our staging and production env).

The quick fix was to deploy the upgrade. Scale the Keycloak StatefulSet down to 0 then scale the StatefulSet back up to your desired amount