[stable/sentry] Fresh install of chart does not work because postgres password

[artefactop]

Describe the bug In a fresh install of the chart, it fails because a timeout. Checking the logs I found the problem.

helm install --name sentry --wait --set service.name=sentry,service.type=ClusterIP,service.externalPort=80,email.host=smtp.mail.com,email.port=1234,email.password=PASS,postgres.enable=true,redis.enable=true stable/sentry --namespace sentry

kubectl logs sentry-db-init

20:26:16 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
20:26:19 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 162, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 36, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 75, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 30, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/usr/local/lib/python2.7/site-packages/south/management/commands/syncdb.py", line 119, in handle_noargs
    self.sync_apps(apps_needing_sync, app_name_to_app_map, options)
  File "/usr/local/lib/python2.7/site-packages/south/management/commands/syncdb.py", line 174, in sync_apps
    syncdb.Command().execute(**options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
    cursor = connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 162, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 95, in _cursor
    cursor = super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 132, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 115, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 115, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  password authentication failed for user "postgres"

kubectl logs sentry-sentry-postgresql-0

2019-11-12 20:38:17.045 GMT [1918] FATAL:  password authentication failed for user "postgres"
2019-11-12 20:38:17.045 GMT [1918] DETAIL:  Password does not match for user "postgres".
    Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"

Version of Helm and Kubernetes:

helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.10-gke.17", GitCommit:"27b48e2b4c9535d185ec945c6a513537e4d116cf", GitTreeState:"clean", BuildDate:"2019-10-21T20:10:26Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart: stable/sentry

What happened: It can not start due to bad configuration of Postgres password for user postgres and database sentry.

What you expected to happen: Install correctly

How to reproduce it (as minimally and precisely as possible): I did a fresh install with the command above

Anything else we need to know:

• Password is correctly created at secrets
• Password is correctly configured at deployments

[dikshant]

I am running into the same issue so I decided to use the slightly older charts at the commit hash: bffe90bfbf85b0e96e0e78773f8ff3e005d84cb9

The commit above has chart version 3.1.1 instead of 3.1.2 in master and has no issues with deployment in Kube 1.12. I haven't taken a look at the changes introduced with the chart version update but I will report my findings and see if I can fix it when I get a chance to take a closer look at it. Right now this might have happened because of changes made to support Kube 1.16 which breaks compatibility with older Kube versions.

[DandyDeveloper]

It looks like this is breaking for upgrades as well, specifically, there's a problem with the chart replacing the existing postgres password.

@artefactop Check to make sure you didn't have an existing secret, if you do, that's probably the issue.

[artefactop]

I tried again and I checked that secrets do not exist before run helm chart but I get the same errors. I there any way I can debug what happens?

[DandyDeveloper]

@artefactop I am assuming you're running through the same thing I did, best thing to do:

Access the psql container, check the env vars and attempt to login - If it works, update the other deployment templates with this password.

Alternatively, change the password in the psql container and update the env vars. The problem with this is you need to do this before the webhooks for migrations kick in - As they will basically be the crux that allows the Sentry web component to run correctly.

[chrisjowen]

I appear to have the same issue, with 3.1.3, well actually maybe not exactly the same as I see:

[2495] DETAIL: User "postgres" has no password assigned.

In the pg container logs.

[seanarnold]

I'm running into this issue too - has anyone found a workaround?

[DandyDeveloper]

@chrisjowen @seanarnold

Can you both check the pod spec and make sure a password is set there? The init scripts should create the password based on that env var.

Did you also make sure the PV/PVC was removed before reinstalling the chart?

[seanarnold]

I used 3.1.1 like @dikshant suggested, and it seems to work now. This is using the currently default GKE version 1.13.11-gke.14.

[thanhtoan1196]

I think this happens because the new password is randomly generated and the old password is still saved in the volume The way to fix this is to not give a random password but specify a password for postgresql, user, redis.

[seanarnold]

@thanhtoan1196 Yeah, it seems like there isn't a way to specify a postgresql password in the Sentry chart though, unless you're hosting Postgres yourself https://github.com/helm/charts/blob/master/stable/sentry/templates/secrets.yaml#L25-L27

Or am I missing something?

[seanarnold]

@thanhtoan1196 - also redis seems to be OK when i perform a subsequent upgrade. I'm not sure why that seems to work, and Postgres fails. Any ideas?

[tckb]

Hi, I am not sure why this was closed, I have the same problem. I setup password for postgres, redis and user. None of which helped.

I still see the same error. Any help?

sentry-sentry-postgresql 2020-03-20 19:43:59.894 GMT [589] FATAL:  password authentication failed for user "postgres"                               │
│ sentry-sentry-postgresql 2020-03-20 19:43:59.894 GMT [589] DETAIL:  Password does not match for user "postgres".

[artefactop]

Check your vpc's. The chart do not clean them between installs so maybe if the first time you tried to install it, the vpc's were created and the password is saved there, if it failed and then you execute the chart again, the vpc's are not reset.

If that's the problem, just delete the vpc's and try again.

El vie., 20 mar. 2020 20:44, Chandra Tungathurthi notifications@github.com escribió:

Hi, I am not sure why this was closed, I have the same problem. I setup password for postgres, redis and user. None of which helped.

I still see the same error. Any help?

sentry-sentry-postgresql 2020-03-20 19:43:59.894 GMT [589] FATAL: password authentication failed for user "postgres" │

│ sentry-sentry-postgresql 2020-03-20 19:43:59.894 GMT [589] DETAIL: Password does not match for user "postgres".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/helm/charts/issues/18822#issuecomment-601880719, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJE2C7HXJCEU5SSOBA3ZNLRIPBTDANCNFSM4JMI3U5A .

[tckb]

@artefactop I am not sure what you mean by vpc? did you mean PVC even in that case it didn't work. I tried multiple times with clean setup.

[evgkrsk]

Same problem here.

[rakesh-007]

@artefactop you can resolve this issue by setting up a fixed password in values.yaml. Use this :+1:
postgresqlPassword: **

[anton-chuhunou]

Same issue here. Fresh install on GKE: django.db.utils.OperationalError: FATAL: password authentication failed for user "postgres"

[dpappmako]

Deleting persistentvolumeclaims seems to help.

[aajuwon]

Deleting PVC works for me

also please set the postgresPassword in the helm chart