search

timescale / tobs

tobs - The Observability Stack for Kubernetes. Easy install of a full observability stack into a k8s cluster with Helm charts.
Apache License 2.0
563 stars 60 forks source link

The promscale-connection secret is not being updated by the Job #554

Closed onprem closed 2 years ago

onprem commented 2 years ago

What happened?

When upgrading from v0.11.0 to v12.1.1, the Job to update the promscale connection secret with correct password is not working as expected. The password had the default value PLACEHOLDER there, and promscale went into CrashLoopBackoff due to incorrect password.

Did you expect to see something different?

The Job works correctly, even in case of upgrades.

How to reproduce it (as minimally and precisely as possible):

Installing v0.11.0 first and then upgrading to v12.1.1 should reproduce this.

abhinavDhulipala commented 2 years ago

Accidentally passed some invalid params to my promscale configs, when I reverted the same thing happened to me. Don't know the fix yet

abhinavDhulipala commented 2 years ago

For some reason, in my values file, my promscale connection secret name was set to "". I simply reverted to the default in the values file and it worked. Don't know if this solves your problem, but it solved mine. 

connectionSecretName: "tobs-promscale-connection"
abhinavDhulipala commented 2 years ago

Ran into this problem again. For some reason the secret gets corrupted if an upgrade fails or cancels sometime? Not quite sure the conditions under which it gets corrupted. But this is how I tracked down the problem. I inspected and the secret as follows.

kubectl get secret -n observability tobs-promscale-connection -o jsonpath="{.data.PROMSCALE_DB_PASSWORD}" | base64 --decode 
PLACEHOLDER

I got PLACEHOLDER. I'm sure somewhere tobs changes this upon connection, but I simply forcefully reset the secret to it's rightful password. i.e I got the pg password

PGPASSWORD_POSTGRES=$(
      kubectl get secret --namespace observability \
        tobs0-credentials \
        -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}" 
    )

and copied the postgres password into data.PROMSCALE_DB_PASSWORD using kubectl edit secret -n observability tobs-promscale-connection and it worked again. Not sure if this is the proper way to work around this, but it seems to work far better than any other alternatives I've seen.

jgerry2002 commented 2 years ago

I've had this same issue. Its always related to helm upgrades. The secret gets corrupted and replaced with PLACEHOLDER every single time. I thought it was something that I was doing and did not get a chance to dig into it, other then note that the "PLACEHOLDER" secret was in there vs the actual password. Thanks for posting a work around. I just edit the secret and replace the value with the correct password and promscale fixes itself.