search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.83k stars 1.76k forks source link

Unable to dispatch invite emails - "EMAIL_USE_TLS/EMAIL_USE_SSL are mutually exclusive, so only set one of those settings to True" #871

Closed Nessworthy closed 3 years ago

Nessworthy commented 3 years ago

Version Information

Version: 21.2.0

Steps to Reproduce

  1. Fresh install of onpremise sentry, using the following configuration: 
    auth.allow-registration: false
beacon.anonymous: true
dsym.cache-path: /data/dsym-cache
filestore.backend: filesystem
filestore.options:
location: /data/files
mail.backend: smtp
mail.from: sentry@parent.com
mail.list-namespace: parent.com
mail.port: 25
mail.use-tls: true
releasefile.cache-path: /data/releasefile-cache
symbolicator.enabled: true
symbolicator.options:
url: http://symbolicator:3021
system.admin-email: adminl@parent.com
system.internal-url-prefix: http://web:9000
system.secret-key: ---
system.url-prefix: https://sentry.parent.com
transaction-events.force-disable-internal-project: true
mail.host: email-smtp.eu-west-1.amazonaws.com
mail.username: ---
mail.password: ---
  2. Create a (super) user account via docker-compose: docker-compose run --rm web sentry createuser
  3. Log in & Invite a new user to sentry via the UI (https://sentry.parent.com/settings/myorganization/members/)

FWIW my email is set up to use AWS SES via SMTP.

Expected Result

User is shown as having been invited. User is sent an invite email.

Actual Result

User is shown as having been invited. User is not sent an invite email.

Logs

web_1                                       | 18:31:26 [INFO] sentry.audit.api: member.invite (ip_address='---' organization_id=1 object_id=8 entry_id=17 actor_label='me@parent.com' actor_id=1)
nginx_1                                     | 192.168.0.36 - - [24/Feb/2021:18:31:26 +0000] "PUT /api/0/organizations/myorganization/invite-requests/8/ HTTP/1.1" 200 318 "https://sentry.parent.com/settings/myorganization/members/requests/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36" "147.147.146.5"
worker_1                                    | Traceback (most recent call last):
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task
worker_1                                    |     R = retval = fun(*args, **kwargs)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__
worker_1                                    |     return self.run(*args, **kwargs)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", line 197, in _inner
worker_1                                    |     reraise(*exc_info)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/_compat.py", line 54, in reraise
worker_1                                    |     raise value
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", line 192, in _inner
worker_1                                    |     return f(*args, **kwargs)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/tasks/base.py", line 46, in _wrapped
worker_1                                    |     result = func(*args, **kwargs)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/tasks/email.py", line 73, in send_email
worker_1                                    |     send_messages([message])
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/email.py", line 404, in send_messages
worker_1                                    |     connection = get_connection(fail_silently=fail_silently)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/email.py", line 437, in get_connection
worker_1                                    |     fail_silently=fail_silently,
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/django/core/mail/__init__.py", line 37, in get_connection
worker_1                                    |     return klass(fail_silently=fail_silently, **kwds)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/django/core/mail/backends/smtp.py", line 34, in __init__
worker_1                                    |     "EMAIL_USE_TLS/EMAIL_USE_SSL are mutually exclusive, so only set "
worker_1                                    | ValueError: EMAIL_USE_TLS/EMAIL_USE_SSL are mutually exclusive, so only set one of those settings to True.
worker_1                                    | 18:31:26 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[ed50b4ac-1e36-4c04-9c0e-de8cd9b66431] raised unexpected: ValueError('EMAIL_USE_TLS/EMAIL_USE_SSL are mutually exclusive, so only set one of those settings to True.',) (data={'hostname': 'e845b9b4b9a1', 'id': 'ed50b4ac-1e36-4c04-9c0e-de8cd9b66431', 'name': 'sentry.tasks.email.send_email', 'exc': "ValueError('EMAIL_USE_TLS/EMAIL_USE_SSL are mutually exclusive, so only set one of those settings to True.',)", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", line 197, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/_compat.py", line 54, in reraise\n    raise value\n  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", line 192, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/tasks/base.py", line 46, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/tasks/email.py", line 73, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.6/site-packages/sentry/utils/email.py", line 404, in send_messages\n    connection = get_connection(fail_silently=fail_silently)\n  File "/usr/local/lib/python3.6/site-packages/sentry/utils/email.py", line 437, in get_connection\n    fail_silently=fail_silently,\n  File "/usr/local/lib/python3.6/site-packages/django/core/mail/__init__.py", line 37, in get_connection\n    return klass(fail_silently=fail_silently, **kwds)\n  File "/usr/local/lib/python3.6/site-packages/django/core/mail/backends/smtp.py", line 34, in __init__\n    "EMAIL_USE_TLS/EMAIL_USE_SSL are mutually exclusive, so only set "\nValueError: EMAIL_USE_TLS/EMAIL_USE_SSL are mutually exclusive, so only set one of those settings to True.\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7febd8f804e0>}", 'description': 'raised unexpected', 'internal': False})

What you saw along the way, e.g.:

Docker logs above. Unfortunately my install logs are filled with another issue as I'm testing using NFS mounts (backed by AWS EFS) which seems to cause enough slowdown to trigger other errors (also concerned with general throughput of this method, but that's what the testing is for!)

▶ Defining variables and helpers ...

▶ Parsing command line ...

▶ Setting up error handling ...

▶ Checking minimum requirements ...

▶ Creating volumes for persistent storage ...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

▶ Ensuring files from examples ...
sentry/sentry.conf.py already exists, skipped creation.    <==== (My build script copies this from the sample file prior to install)
sentry/config.yml already exists, skipped creation.
Creating sentry/requirements.txt...
Creating symbolicator/config.yml...
Creating relay/config.yml...

▶ Generating secret key ...

▶ Replacing TSDB ...

▶ Fetching and updating Docker images ...
--no-ansi option is deprecated and will be removed in future versions.
Some service image(s) must be built from source by running:
    docker-compose build symbolicator-cleanup sentry-cleanup snuba-cleanup
21.2.0: Pulling from getsentry/sentry
Digest: sha256:24442be169e58b99fae7210884b9a9c46671f3002bda9e19f4bd3b172dc74133
Status: Image is up to date for getsentry/sentry:21.2.0
docker.io/getsentry/sentry:21.2.0

▶ Building and tagging Docker images ...

--no-ansi option is deprecated and will be removed in future versions.
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
geoipupdate uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
snuba-transactions-consumer uses an image, skipping
snuba-replacer uses an image, skipping
snuba-subscription-consumer-events uses an image, skipping
snuba-subscription-consumer-transactions uses an image, skipping
symbolicator uses an image, skipping
web uses an image, skipping
cron uses an image, skipping
worker uses an image, skipping
ingest-consumer uses an image, skipping
post-process-forwarder uses an image, skipping
subscription-consumer-events uses an image, skipping
subscription-consumer-transactions uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Building snuba-cleanup
Sending build context to Docker daemon  3.072kB

Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 60b95aebecb1
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 0d4a9eca94fe
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://security.debian.org/debian-security buster/updates/main amd64 Packages [267 kB]
Get:3 http://deb.debian.org/debian buster InRelease [122 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [9504 B]
Fetched 8422 kB in 3s (3001 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  lsb-base sensible-utils
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  cron lsb-base sensible-utils
0 upgraded, 3 newly installed, 0 to remove and 3 not upgraded.
Need to get 143 kB of archives.
After this operation, 383 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 sensible-utils all 0.0.12 [15.8 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 lsb-base all 10.2019051400 [28.4 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 cron amd64 3.0pl1-134+deb10u1 [99.0 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 143 kB in 0s (9930 kB/s)
Selecting previously unselected package sensible-utils.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 6840 files and directories currently installed.)
Preparing to unpack .../sensible-utils_0.0.12_all.deb ...
Unpacking sensible-utils (0.0.12) ...
Selecting previously unselected package lsb-base.
Preparing to unpack .../lsb-base_10.2019051400_all.deb ...
Unpacking lsb-base (10.2019051400) ...
Selecting previously unselected package cron.
Preparing to unpack .../cron_3.0pl1-134+deb10u1_amd64.deb ...
Unpacking cron (3.0pl1-134+deb10u1) ...
Setting up lsb-base (10.2019051400) ...
Setting up sensible-utils (0.0.12) ...
Setting up cron (3.0pl1-134+deb10u1) ...
Adding group `crontab' (GID 101) ...
Done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 0d4a9eca94fe
 ---> e039adf248a0
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> 946d03ab3a87
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in 7242a1fc90c1
Removing intermediate container 7242a1fc90c1
 ---> 88b9a37e2662
Successfully built 88b9a37e2662
Successfully tagged snuba-cleanup-onpremise-local:latest
Building symbolicator-cleanup
Sending build context to Docker daemon  3.072kB

Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 77d674e404a3
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 520ab2658eec
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://deb.debian.org/debian stretch-updates InRelease [93.6 kB]
Get:3 http://deb.debian.org/debian stretch Release [118 kB]
Get:4 http://deb.debian.org/debian stretch Release.gpg [2410 B]
Get:5 http://security.debian.org/debian-security stretch/updates InRelease [53.0 kB]
Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages [2596 B]
Get:7 http://deb.debian.org/debian stretch/main amd64 Packages [7080 kB]
Get:8 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [656 kB]
Fetched 8006 kB in 1s (5492 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  exim4 | postfix | mail-transport-agent
The following NEW packages will be installed:
  cron
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 95.4 kB of archives.
After this operation, 257 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 cron amd64 3.0pl1-128+deb9u1 [95.4 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 95.4 kB in 0s (0 B/s)
Selecting previously unselected package cron.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 6661 files and directories currently installed.)
Preparing to unpack .../cron_3.0pl1-128+deb9u1_amd64.deb ...
Unpacking cron (3.0pl1-128+deb9u1) ...
Setting up cron (3.0pl1-128+deb9u1) ...
Adding group `crontab' (GID 101) ...
Done.
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 520ab2658eec
 ---> 6ba67dcb88ee
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> 1aab21e5ebfc
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in a9447b12f946
Removing intermediate container a9447b12f946
 ---> 7cb4513a1cd4
Successfully built 7cb4513a1cd4
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building sentry-cleanup
Sending build context to Docker daemon  3.072kB

Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> e1f6e507005f
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 73ec4cbfb4bf
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [267 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [9504 B]
Fetched 8422 kB in 1s (5717 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  lsb-base sensible-utils
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  cron lsb-base sensible-utils
0 upgraded, 3 newly installed, 0 to remove and 4 not upgraded.
Need to get 143 kB of archives.
After this operation, 383 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 sensible-utils all 0.0.12 [15.8 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 lsb-base all 10.2019051400 [28.4 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 cron amd64 3.0pl1-134+deb10u1 [99.0 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 143 kB in 0s (9676 kB/s)
Selecting previously unselected package sensible-utils.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 11935 files and directories currently installed.)
Preparing to unpack .../sensible-utils_0.0.12_all.deb ...
Unpacking sensible-utils (0.0.12) ...
Selecting previously unselected package lsb-base.
Preparing to unpack .../lsb-base_10.2019051400_all.deb ...
Unpacking lsb-base (10.2019051400) ...
Selecting previously unselected package cron.
Preparing to unpack .../cron_3.0pl1-134+deb10u1_amd64.deb ...
Unpacking cron (3.0pl1-134+deb10u1) ...
Setting up lsb-base (10.2019051400) ...
Setting up sensible-utils (0.0.12) ...
Setting up cron (3.0pl1-134+deb10u1) ...
Adding group `crontab' (GID 101) ...
Done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 73ec4cbfb4bf
 ---> c9e990503e1d
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> 5fc4b488150e
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in 30b987346c3f
Removing intermediate container 30b987346c3f
 ---> a63968caa4d3
Successfully built a63968caa4d3
Successfully tagged sentry-cleanup-onpremise-local:latest

Docker images built.

▶ Turning things off ...
--no-ansi option is deprecated and will be removed in future versions.
Removing network onpremise_default
Network onpremise_default not found.
--no-ansi option is deprecated and will be removed in future versions.
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.

▶ Setting up Zookeeper ...
--no-ansi option is deprecated and will be removed in future versions.
Creating network "sentry_onpremise_default" with the default driver
Creating volume "sentry_onpremise_sentry-secrets" with default driver
Creating volume "sentry_onpremise_sentry-smtp" with default driver
Creating volume "sentry_onpremise_sentry-zookeeper-log" with default driver
Creating volume "sentry_onpremise_sentry-kafka-log" with default driver
Creating volume "sentry_onpremise_sentry-smtp-log" with default driver
Creating volume "sentry_onpremise_sentry-clickhouse-log" with default driver
Creating sentry_onpremise_zookeeper_run ...
Creating sentry_onpremise_zookeeper_run ... done
--no-ansi option is deprecated and will be removed in future versions.
Creating sentry_onpremise_zookeeper_run ...
Creating sentry_onpremise_zookeeper_run ... done
--no-ansi option is deprecated and will be removed in future versions.
Creating sentry_onpremise_zookeeper_run ...
Creating sentry_onpremise_zookeeper_run ... done

▶ Bootstrapping and migrating Snuba ...
--no-ansi option is deprecated and will be removed in future versions.
Creating sentry_onpremise_zookeeper_1 ...
Creating sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_redis_1      ...
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ...
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_kafka_1      ... done
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --no-migrate --force
+ set gosu snuba snuba bootstrap --no-migrate --force
+ exec gosu snuba snuba bootstrap --no-migrate --force
%3|1614190958.628|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 1ms in state CONNECT)
2021-02-24 18:22:39,635 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1614190959.647|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 12ms in state CONNECT, 1 identical error(s) suppressed)
%3|1614190960.648|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1614190961.655|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-02-24 18:22:41,657 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1614190962.660|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1614190963.660|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-02-24 18:22:43,661 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2021-02-24 18:22:45,820 Failed to create topic events
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'events' already exists."}
2021-02-24 18:22:45,821 Failed to create topic snuba-commit-log
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'snuba-commit-log' already exists."}
2021-02-24 18:22:45,822 Failed to create topic ingest-sessions
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'ingest-sessions' already exists."}
2021-02-24 18:22:45,822 Failed to create topic outcomes
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'outcomes' already exists."}
2021-02-24 18:22:45,822 Failed to create topic cdc
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'cdc' already exists."}
2021-02-24 18:22:45,823 Failed to create topic event-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'event-replacements' already exists."}
--no-ansi option is deprecated and will be removed in future versions.
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
+ '[' m = - ']'
+ snuba migrations --help
+ set -- snuba migrations migrate --force
+ set gosu snuba snuba migrations migrate --force
+ exec gosu snuba snuba migrations migrate --force
Failed to connect to clickhouse:9000
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 256, in connect
    return self._init_connection(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 226, in _init_connection
    self.socket = self._create_socket(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 221, in _create_socket
    raise err
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 212, in _create_socket
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Failed to connect to clickhouse:9000
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 256, in connect
    return self._init_connection(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 226, in _init_connection
    self.socket = self._create_socket(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 221, in _create_socket
    raise err
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 212, in _create_socket
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Connection to Clickhouse cluster clickhouse:9000 failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/clickhouse/native.py", line 78, in execute
    result: Sequence[Any] = conn.execute(
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 199, in execute
    self.connection.force_connect()
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 180, in force_connect
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 277, in connect
    raise err
clickhouse_driver.errors.NetworkError: Code: 210. Connection refused (None:None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/snuba/snuba/migrations/connect.py", line 30, in check_clickhouse_connections
    check_clickhouse(clickhouse)
  File "/usr/src/snuba/snuba/migrations/connect.py", line 49, in check_clickhouse
    ver = clickhouse.execute("SELECT version()")[0][0]
  File "/usr/src/snuba/snuba/clickhouse/native.py", line 93, in execute
    raise ClickhouseError(e.code, e.message) from e
snuba.clickhouse.errors.ClickhouseError: [210] Connection refused (None:None)
Failed to connect to clickhouse:9000
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 256, in connect
    return self._init_connection(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 226, in _init_connection
    self.socket = self._create_socket(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 221, in _create_socket
    raise err
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 212, in _create_socket
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

[ REPEATS FOR A WHILE ]

1
An error occurred, caught SIGERR on line 283
Cleaning up...
Nessworthy commented 3 years ago

Solution found!

Added mail.use-ssl to my config.yml:

mail.use-tls: true
mail.use-ssl: false

And in the logs:

web_1                                       | 09:16:36 [INFO] sentry.mail: mail.queued (message_type='organization.invite' organization_id=1 message_id='<20210225091635.24.89306@parent.com>' message_to=('john.smith@parent.com',))
...
worker_1                                    | 09:16:36 [INFO] sentry.mail: mail.sent (message_id='<20210225091635.24.89306@parent.com>' size=4988)