[CLOUD-2453][KEYCLOAK-7098] Add database connectivity check

[iankko]

prior launching the RH-SSO server when deploying the RH-SSO pod from persistent DB templates

Signed-off-by: Jan Lieskovsky jlieskov@redhat.com

[iankko]

@rcernich @Crumby PTAL

Thank you, Jan

[luck3y]

One thing I've noticed is the upstream OS mysql template uses wait-for-ready, where we don't seem to have that in our templates.

https://github.com/openshift/origin/blob/master/examples/db-templates/mysql-persistent-template.json#L84

[rcernich]

Sounds like a simpler fix maybe?

[iankko]

@luck3y Thanks, Ken! Noticed that one, when searching how this could be done. Seems to have been introduced starting from v3.7. Tried, but it wouldn't help me in this specific case (the pods are still deployed from both sso and sso-mysql deployments at the same time, still leading to the JPA RH-SSO exception, without RH-SSO pod restart).

But seems to be something useful to apply to our templates too. IIUR when using that annotation, if the build failed, the template instantiation will fail immediately (thus the namespace might not be polluted by objects [services, routes], which wouldn't be working anyway).

[rcernich]

@iankko, does something need to be done to the liveness probe to have it fail if sso isn't properly initialized? That's the way it should behave (basically, if sso isn't working, liveness should fail).

As a side note, it would be interesting if we could find a way to fix this within the JPA initialization code, e.g. have it wait until the db is available. Another option would be to manually install the schema, which might be accomplished through an init container, e.g. init container checks the db to see if the schema is installed and, if not, installs the schema and lets the process proceed; if not, just fails and the pod is rescheduled. (I think the desired behavior is to have the pod bounce until the environment is stable.)

[iankko]

@rcernich The readiness probe correctly fails currently, and is followed with container restart (4-times in 2 minutes). But the issue is we are starting RH-SSO pod, even when we know, it can't work properly (since it takes 2-3 minutes till MySQL pod is initialized properly).

For the JPA initialization code, tried to set up allocation-retry, allocation-retry-wait-millis, and blocking-timeout-millis / blocking-timeout-wait-millis timeout properties on the datasource, without luck so far (even having them set, no change in RH-SSO pod behaviour). But will explore the options further yet.

[iankko]

Closing (will submit different proposal).