nginx-server-block can't be found after upgrade

renich commented 4 years ago

I had a pretty recent installation.

cs-sentry   cs-sentry   2           2020-08-05 01:26:10.342353484 +0000 UTC deployed    sentry-4.7.2    20.7.0

And upgraded to:

cs-sentry   cs-sentry   3           2020-08-19 14:43:15.664040364 +0000 UTC deployed    sentry-5.0.0    20.7.2

And nginx and nginx and one of the web pods are failing.

Also, I am seeing these events in the namespace:

0s          Warning   FailedMount              pod/cs-sentry-nginx-cdd44d49b-zq8jt                       MountVolume.SetUp failed for volume "nginx-server-block" : configmap "sentry-nginx" not found

The upgrade process was as follows:

helm repo update; 
helm upgrade --install -n cs-sentry -f ~/src/sentry/my-values.yaml cs-sentry sentry/sentry

renich commented 4 years ago

Another thing, here's my configmap list:

root@k8s0:~# kubectl -n cs-sentry get configmap
NAME                            DATA   AGE
cs-sentry-clickhouse-config     1      8d
cs-sentry-clickhouse-metrica    1      8d
cs-sentry-clickhouse-users      1      8d
cs-sentry-kafka-scripts         1      8d
cs-sentry-nginx                 1      36m
cs-sentry-nginx-server-block    1      36m
cs-sentry-rabbitmq              2      8d
cs-sentry-relay                 1      36m
cs-sentry-sentry                2      8d
cs-sentry-sentry-redis          3      8d
cs-sentry-sentry-redis-health   6      8d
cs-sentry-snuba                 1      8d

J0sh0nat0r commented 4 years ago

@renich This is a known issue in the current version, you should be able to fix it by setting nginx.existingServerBlockConfigmap to cs-sentry-nginx

renich commented 4 years ago

I had to add that to the my-values.yaml file I have. Also, I had to manually delete both web pods in order for the volume to be mounted.

Another thing. I can't login after this. It keeps complaining about CSRF being wrong. I've cleared all cookies and all. No idea.

I dunno if this is the right place to post all of this. Let's just say that these are problems caused by upgrading! ;D

J0sh0nat0r commented 4 years ago

@renich That'll be #153, it should be fixed soon. If you're using the NGINX ingress controller you should set nginx.enabled to false, or if you're using Traefik I can provide the IngressRoute we have configured

renich commented 4 years ago

That did it @J0sh0nat0r. Thanks a lot.

fabriciols commented 4 years ago

i had the same on a new frash install, using this cmd:

$ cat sentry.yaml 
nginx.existingServerBlockConfigmap: sentry-prod-ngnix

helm install sentry-prod sentry/sentry -f sentry.yaml

and the nginx pod doesnt get up

│   Type     Reason       Age                    From                                      Message                                                                                                                                                                                                                                                                        │
│   ----     ------       ----                   ----                                      -------                                                                                                                                                                                                                                                                        │
│   Normal   Scheduled    <unknown>              default-scheduler                         Successfully assigned default/sentry-prod-nginx-649b45779d-rpqlz to aks-npprod3-10241778-vmss000002                                                                                                                                                                            │
│   Warning  FailedMount  2m52s (x3 over 7m24s)  kubelet, aks-npprod3-10241778-vmss000002  Unable to attach or mount volumes: unmounted volumes=[nginx-server-block], unattached volumes=[nginx-server-block-paths nginx-server-block default-token-p2wbf]: timed out waiting for the condition                                                                           │
│   Warning  FailedMount  73s (x12 over 9m27s)   kubelet, aks-npprod3-10241778-vmss000002  MountVolume.SetUp failed for volume "nginx-server-block" : configmap "sentry-nginx" not found                                                                                                                                                                                  │
│   Warning  FailedMount  35s                    kubelet, aks-npprod3-10241778-vmss000002  Unable to attach or mount volumes: unmounted volumes=[nginx-server-block], unattached volumes=[nginx-server-block default-token-p2wbf nginx-server-block-paths]: timed out waiting for the condition                                                                           │

Also , the init-db pod is throwing some errors

│ 16:40:27 [INFO] sentry.plugins.github: apps-not-configured                                                                                                                                                                                                                                                                                                              │
│ Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                      │
│   File "/usr/local/bin/sentry", line 8, in <module>                                                                                                                                                                                                                                                                                                                     │
│     sys.exit(main())                                                                                                                                                                                                                                                                                                                                                    │
│   File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 166, in main                                                                                                                                                                                                                                                                            │
│     cli(prog_name=get_prog(), obj={}, max_content_width=100)                                                                                                                                                                                                                                                                                                            │
│   File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__                                                                                                                                                                                                                                                                                    │
│     return self.main(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                   │
│   File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main                                                                                                                                                                                                                                                                                        │
│     rv = self.invoke(ctx)                                                                                                                                                                                                                                                                                                                                               │
│   File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke                                                                                                                                                                                                                                                                                     │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                                                                                                             │
│   File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke                                                                                                                                                                                                                                                                                      │
│     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                                                                                                                                      │
│   File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke                                                                                                                                                                                                                                                                                      │
│     return callback(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                    │
│   File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func                                                                                                                                                                                                                                                                               │
│     return f(get_current_context(), *args, **kwargs)                                                                                                                                                                                                                                                                                                                    │
│   File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner                                                                                                                                                                                                                                                                          │
│     return ctx.invoke(f, *args, **kwargs)                                                                                                                                                                                                                                                                                                                               │
│   File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke                                                                                                                                                                                                                                                                                      │
│     return callback(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                    │
│   File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func                                                                                                                                                                                                                                                                               │
│     return f(get_current_context(), *args, **kwargs)                                                                                                                                                                                                                                                                                                                    │
│   File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 174, in upgrade                                                                                                                                                                                                                                                                 │
│     _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)                                                                                                                                                                                                                                                                                          │
│   File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 121, in _upgrade                                                                                                                                                                                                                                                                │
│     _migrate_from_south(verbosity)                                                                                                                                                                                                                                                                                                                                      │
│   File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 93, in _migrate_from_south                                                                                                                                                                                                                                                      │
│     if not _has_south_history(connection):                                                                                                                                                                                                                                                                                                                              │
│   File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 78, in _has_south_history                                                                                                                                                                                                                                                       │
│     cursor = connection.cursor()                                                                                                                                                                                                                                                                                                                                        │
│   File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor                                                                                                                                                                                                                                                                    │
│     return self._cursor()                                                                                                                                                                                                                                                                                                                                               │
│   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner                                                                                                                                                                                                                                                                     │
│     return func(self, *args, **kwargs)                                                                                                                                                                                                                                                                                                                                  │
│   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 97, in _cursor                                                                                                                                                                                                                                                                         │
│     return super(DatabaseWrapper, self)._cursor()                                                                                                                                                                                                                                                                                                                       │
│   File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor                                                                                                                                                                                                                                                                   │
│     self.ensure_connection()                                                                                                                                                                                                                                                                                                                                            │
│   File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection                                                                                                                                                                                                                                                         │
│     self.connect()                                                                                                                                                                                                                                                                                                                                                      │
│   File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__                                                                                                                                                                                                                                                                                │
│     six.reraise(dj_exc_type, dj_exc_value, traceback)                                                                                                                                                                                                                                                                                                                   │
│   File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection                                                                                                                                                                                                                                                         │
│     self.connect()                                                                                                                                                                                                                                                                                                                                                      │
│   File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect                                                                                                                                                                                                                                                                   │
│     self.connection = self.get_new_connection(conn_params)                                                                                                                                                                                                                                                                                                              │
│   File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection                                                                                                                                                                                                                                                  │
│     connection = Database.connect(**conn_params)                                                                                                                                                                                                                                                                                                                        │
│   File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 127, in connect                                                                                                                                                                                                                                                                              │
│     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)                                                                                                                                                                                                                                                                                              │
│ django.db.utils.OperationalError: FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                                                            │
│ stream closed                                                                                                                                                                                                                                                                                                                                                           │

and postgres pod

│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:12.081 GMT [1309] FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                             │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:12.081 GMT [1309] DETAIL:  Password does not match for user "postgres".                                                                                                                                                                                                                                                  │
│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:12.084 GMT [1311] FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                             │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:12.084 GMT [1311] DETAIL:  Password does not match for user "postgres".                                                                                                                                                                                                                                                  │
│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:12.088 GMT [1312] FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                             │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:12.088 GMT [1312] DETAIL:  Password does not match for user "postgres".                                                                                                                                                                                                                                                  │
│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:13.669 GMT [1320] FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                             │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:13.669 GMT [1320] DETAIL:  Password does not match for user "postgres".                                                                                                                                                                                                                                                  │
│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:18.177 GMT [1321] FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                             │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:18.177 GMT [1321] DETAIL:  Password does not match for user "postgres".                                                                                                                                                                                                                                                  │
│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:18.450 GMT [1322] FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                             │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:18.450 GMT [1322] DETAIL:  Password does not match for user "postgres".                                                                                                                                                                                                                                                  │
│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:23.687 GMT [1335] FATAL:  password authentication failed for user "postgres"                                                                                                                                                                                                                                             │
│ sentry-prod-sentry-postgresql 2020-09-14 16:50:23.687 GMT [1335] DETAIL:  Password does not match for user "postgres".                                                                                                                                                                                                                                                  │
│ sentry-prod-sentry-postgresql     Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"                                                                                                                                                                                                                         │

(should i open a new issue?)

renich commented 4 years ago

@fabriciols did you open another issue? I'm getting the same thing after a recent update.

0s          Warning   FailedMount          pod/cs-sentry-web-6dcc99bd89-5mhxm                            Unable to attach or mount volumes: unmounted volumes=[config sentry-data default-token-cq675], unattached volumes=[config sentry-data default-token-cq675]: timed out waiting for the condition

sentry-kubernetes / charts

nginx-server-block can't be found after upgrade #154