search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.94k stars 1.78k forks source link

SMTP TLS/SSL not working anymore #2745

Open StefanIGit opened 10 months ago

StefanIGit commented 10 months ago

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

upgrade to 24.1.0 from 23.8.0 from sentry/config.yml

mail.backend: 'smtp'
mail.host: mail6.server.tld
mail.password: 'secret'
mail.username: 'info@server.tld'
mail.from: 'sentry@server.tld'
mail.use-ssl: true
mail.port: 587

it worked before the settings are correct since they work fine with Thunderbird

The server requires "STARTTLS" it is try mail.use-ssl: false or mail.use-tls: true or mail.use-ssl: false mail.use-tls: false I get error like SMTPServerDisconnected('Connection unexpectedly closed: timed out') and SMTPNotSupportedError('SMTP AUTH extension not supported by server.')

Expected Result

sending email (invite/pwrest) successfully

Actual Result

errors in log

sentry-self-hosted-worker-1                                        | 10:15:48 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[6b605357-fb39-4788-a43d-8e68b58d49cf] raised unexpected: SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1007)') (data={'hostname': 'celery@a75b7517a419', 'id': '6b605357-fb39-4788-a43d-8e68b58d49cf', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1007)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise\n    raise value\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages\n    sent = connection.send_messages(messages)\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages\n    new_conn_created = self.open()\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 62, in open\n    self.connection = self.connection_class(self.host, self.port, **connection_params)\n  File "/usr/local/lib/python3.10/smtplib.py", line 1050, in __init__\n    SMTP.__init__(self, host, port, local_hostname, timeout,\n  File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__\n    (code, msg) = self.connect(host, port)\n  File "/usr/local/lib/python3.10/smtplib.py", line 341, in connect\n    self.sock = self._get_socket(host, port, self.timeout)\n  File "/usr/local/lib/python3.10/smtplib.py", line 1057, in _get_socket\n    new_socket = self.context.wrap_socket(new_socket,\n  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket\n    return self.sslsocket_class._create(\n  File "/usr/local/lib/python3.10/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.10/ssl.py", line 1375, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1007)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fe621961990>, '__start_time': 1706523347.505079}", 'description': 'raised unexpected', 'internal': False})

Product Area

Settings

Link

No response

DSN

No response

Version

24.1.0

getsantry[bot] commented 10 months ago

Assigning to @getsentry/support for routing ⏲️

hubertdeng123 commented 10 months ago

Is your server using TLS as well? We've also upgraded successfully without encountering this issue, so I don't think this is a general problem.

Leask commented 9 months ago

same here:

worker-1                                        | smtplib.SMTPServerDisconnected: Connection unexpectedly closed: timed out
azaslavsky commented 9 months ago

There was a breaking change at 23.11.0 that removed a server that was not on the default SMTP path. Is it possible that either of you were using it?

chris-laack commented 9 months ago

We have the same problem since version sentry 24.*. However, it is due to a faulty server certificate. This was not noticed before the changeover to Django Mail Vers. 4.2, as the host entries were not checked. Since the new Django version this is now necessary.

Error: SSL: CERTIFICATE_VERIFY_FAILED This is due to the ssl_context.check_hostname field in the django.core.mail.backends.smtp.py file being set to True by default, starting from Django 4.2.

Unfortunately I have not found a way to work around this using skip ssl validation. In our case it only helps that our mail admin issues a valid server certificate.

hubertdeng123 commented 9 months ago

@chris-laack Thanks for your input. Does that help @StefanIGit ?

AwiOnline commented 8 months ago

I have the same problem on 23.11.2 It worked before upgrade from 23.8.0

hubertdeng123 commented 8 months ago

What is the error message you are seeing @AwiOnline?

AwiOnline commented 8 months ago

Here is the error message: "SMTPServerDisconnected('Connection unexpectedly closed: timed out')"

hubertdeng123 commented 8 months ago

Are you also using thunderbird? I am unable to reproduce this and we are using sendgrid

AwiOnline commented 8 months ago

Are you also using thunderbird? I am unable to reproduce this and we are using sendgrid

I'm using Gmail as an SMTP relay

hubertdeng123 commented 8 months ago

I'm going to keep this open to see if there is further input from the community, as I'm afraid I'm not able to reproduce this issue.

michaelkuty commented 7 months ago

same for me, struggling to set mailgun as SMTP, 

mail.host: 'smtp.mailgun.org'
mail.port: 587
mail.username: 'sentry@domain.com'
mail.password: 'secret'
mail.use-tls: true
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
worker-1  |     R = retval = fun(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner
worker-1  |     reraise(*exc_info)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise
worker-1  |     raise value
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner
worker-1  |     return f(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
worker-1  |     return self.run(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override
worker-1  |     return original_method(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped
worker-1  |     result = func(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email
worker-1  |     send_messages([message])
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages
worker-1  |     sent = connection.send_messages(messages)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages
worker-1  |     new_conn_created = self.open()
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open
worker-1  |     self.connection = self.connection_class(
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__
worker-1  |     (code, msg) = self.connect(host, port)
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect
worker-1  |     (code, msg) = self.getreply()
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply
worker-1  |     raise SMTPServerDisconnected("Connection unexpectedly closed")
worker-1  | smtplib.SMTPServerDisconnected: Connection unexpectedly closed
worker-1  | 14:43:10 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[997c31df-908e-4401-92df-b60cfcd3c453] raised unexpected: SMTPServerDisconnected('Connection unexpectedly closed') (data={'hostname': 'celery@58c1d45811e5', 'id': '997c31df-908e-4401-92df-b60cfcd3c453', 'name': 'sentry.tasks.email.send_email', 'exc': "SMTPServerDisconnected('Connection unexpectedly closed')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise\n    raise value\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages\n    sent = connection.send_messages(messages)\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages\n    new_conn_created = self.open()\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open\n    self.connection = self.connection_class(\n  File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__\n    (code, msg) = self.connect(host, port)\n  File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect\n    (code, msg) = self.getreply()\n  File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply\n    raise SMTPServerDisconnected("Connection unexpectedly closed")\nsmtplib.SMTPServerDisconnected: Connection unexpectedly closed\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fe2f2523820>, '__start_time': 1713969790.239359}", 'description': 'raised unexpected', 'internal': False})
azaslavsky commented 7 months ago

@michaelkuty are you seeing the CERITIFICATE_VERIFY_FAILED error as well?

michaelkuty commented 7 months ago

@michaelkuty are you seeing the CERITIFICATE_VERIFY_FAILED error as well?

no, I dont see anything else then this error, this is also reason why this is kinda strange because Ive installed like 3 sentry-self hosted last year and without any issue and currently there is no additional debug info just this error and that is all

also I forgot to mention sentry version Sentry 24.1.1

lcsvcn commented 6 months ago

I am at sentry 24.4.2.

I am having issues with sendgrid SMTP:

Screenshot 2024-05-08 at 15 21 39 Screenshot 2024-05-08 at 15 18 53

d">

18:18:35 [INFO] sentry.superuser: superuser.request (url='http://localhost/api/0/internal/mail/' method='POST' ip_address='172.18.0.1' user_id=1)
18:18:41 [INFO] sentry.superuser: superuser.request (url='http://localhost/api/0/internal/mail/' method='POST' ip_address='172.18.0.1' user_id=1)
18:18:43 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.015624046325683594 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:18:53 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.01541447639465332 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:03 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016051292419433594 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:10 [INFO] sentry.superuser: superuser.request (url='http://localhost/api/0/internal/mail/' method='POST' ip_address='172.18.0.1' user_id=1)
18:19:13 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.015586614608764648 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:23 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016732454299926758 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
^[[B18:19:33 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016119956970214844 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:43 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.01533365249633789 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:46 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.internal.mail.InternalMailEndpoint' response=500 user_id='1' is_app='False' token_type='None' is_frontend_request='True' organization_id='None' auth_id='None' path='/api/0/internal/mail/' caller_ip='172.18.0.1' user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=120.13616347312927 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:46 [ERROR] django.request: Internal Server Error: /api/0/internal/mail/ (status_code=500 request=<WSGIRequest: POST '/api/0/internal/mail/'>)
lcsvcn commented 6 months ago

btw, the issue is exclusive with STARTTLS, if I use SSL works fine:

Screenshot 2024-05-08 at 15 29 26 Screenshot 2024-05-08 at 15 30 39
MrKoopie commented 5 months ago

same for me, struggling to set mailgun as SMTP,

mail.host: 'smtp.mailgun.org'
mail.port: 587
mail.username: 'sentry@domain.com'
mail.password: 'secret'
mail.use-tls: true
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
worker-1  |     R = retval = fun(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner
worker-1  |     reraise(*exc_info)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise
worker-1  |     raise value
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner
worker-1  |     return f(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
worker-1  |     return self.run(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override
worker-1  |     return original_method(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped
worker-1  |     result = func(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email
worker-1  |     send_messages([message])
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages
worker-1  |     sent = connection.send_messages(messages)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages
worker-1  |     new_conn_created = self.open()
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open
worker-1  |     self.connection = self.connection_class(
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__
worker-1  |     (code, msg) = self.connect(host, port)
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect
worker-1  |     (code, msg) = self.getreply()
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply
worker-1  |     raise SMTPServerDisconnected("Connection unexpectedly closed")
worker-1  | smtplib.SMTPServerDisconnected: Connection unexpectedly closed
worker-1  | 14:43:10 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[997c31df-908e-4401-92df-b60cfcd3c453] raised unexpected: SMTPServerDisconnected('Connection unexpectedly closed') (data={'hostname': 'celery@58c1d45811e5', 'id': '997c31df-908e-4401-92df-b60cfcd3c453', 'name': 'sentry.tasks.email.send_email', 'exc': "SMTPServerDisconnected('Connection unexpectedly closed')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise\n    raise value\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages\n    sent = connection.send_messages(messages)\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages\n    new_conn_created = self.open()\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open\n    self.connection = self.connection_class(\n  File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__\n    (code, msg) = self.connect(host, port)\n  File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect\n    (code, msg) = self.getreply()\n  File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply\n    raise SMTPServerDisconnected("Connection unexpectedly closed")\nsmtplib.SMTPServerDisconnected: Connection unexpectedly closed\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fe2f2523820>, '__start_time': 1713969790.239359}", 'description': 'raised unexpected', 'internal': False})

Seeing the date of your post, you might be hit by the requirement to configure the IP in the Allow list. Mailgun has introduced this in April.

yildizozgur commented 5 months ago

Hi We have the same issue on 24.5.0.

13:20:36 [ERROR] django.request: Internal Server Error: /api/0/internal/mail/ (status_code=500 request=<WSGIRequest: POST '/api/0/internal/mail/'>)

I can make an SSL handshake on container. Appliction could not complete SSL verification.

azaslavsky commented 5 months ago

Is there no other error information? A 500 error alone doesn't give us much to go on, and usually the logs contain more information about the failed SSL exchange.

yildizozgur commented 5 months ago

Hi, This is my settings:

  backend: smtp
  useTls: true
  useSsl: false
  host: my.mailserver.com:
  from: xxxx
  port: 25
  username: xxxx
  existingSecret: xxxx

I can make a success handshake on the pod.


I have no name!@sentry-qa-worker-xxxxx:/$ openssl s_client -starttls smtp -connect my.mailserver.com:25
CONNECTED(00000003)
depth=2 C = XX, O = XXXX, OU = xxxx.com, CN = XXXX Root CA 
verify error:num=19:self-signed certificate in certificate chain
verify return:1
.....
verify return:1
.....
verify return:1
......
verify return:1
---
Certificate chain
 0 s:O = xxxxxx...............
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3756 bytes and written 433 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
250 HELP

Here is the all worker certificate settings and pod errors:

/$ more /etc/sentry/sentry.conf.py
SENTRY_SDK_CONFIG = {
    'ca_certs': '/etc/pki/ca-trust/custom/ca.crt',
    "release": sentry.__build__,
    "environment": ENVIRONMENT,
    "in_app_include": ["sentry", "sentry_plugins"],
    "debug": True,
    "send_default_pii": True,
    "auto_enabling_integrations": False,
}

$ env | grep BUNDLE
REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/custom/ca.crt
06:30:00 [INFO] sentry.tasks.auto_ongoing_issues: auto_transition_issues_new_to_ongoing started (first_seen_lte=1717396200 first_seen_lte_datetime=datetime.datetime(2024, 6, 3, 6, 30, tzinfo=datetime.timezone.utc))
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:21 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[0c3eedcd-5946-4f52-a156-f4faa50c32f8] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': '0c3eedcd-5946-4f52-a156-f4faa50c32f8', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f078855f850>, '__start_time': 1718001080.878142}", 'description': 'raised unexpected', 'internal': False})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:21 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[dd8c1177-6176-446a-820b-2f68dff3c94f] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': 'dd8c1177-6176-446a-820b-2f68dff3c94f', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f07907cb5d0>, '__start_time': 1718001081.443496}", 'description': 'raised unexpected', 'internal': False})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:22 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[b2c63c9b-8536-49ea-be25-5e99672168fb] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': 'b2c63c9b-8536-49ea-be25-5e99672168fb', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f0788594890>, '__start_time': 1718001081.931252}", 'description': 'raised unexpected', 'internal': False})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:22 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[d07059e3-fb47-4176-b158-db1587eadbd8] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': 'd07059e3-fb47-4176-b158-db1587eadbd8', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f078830e110>, '__start_time': 1718001082.394444}", 'description': 'raised unexpected', 'internal': False})
hubertdeng123 commented 5 months ago

@yildizozgur This may be a similar issue, but also not considering you're using sentry-kubernetes, which isn't supported here.

yildizozgur commented 5 months ago

Hi @hubertdeng123 , yes We are running on K8s. But the problem is related on Application, it is not related to the environment. In the pod we can do SSL handshake successfully.

marbon87 commented 5 months ago

I am having the same problem without using k8s.

hubertdeng123 commented 5 months ago

Thanks for reporting, I'm going to backlog this item for now.

otbutz commented 4 months ago 
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)

I had the same problem after following the guide in https://develop.sentry.dev/self-hosted/custom-ca-roots/. Please make sure that your certificate has the expected format and file extension. See https://github.com/getsentry/develop/pull/1334

yildizozgur commented 1 month ago

Hi @otbutz , I have already set *.crt file extention. It still get same error.

[ERROR] django.request: Internal Server Error: /api/0/internal/mail/ (status_code=500 request=<WSGIRequest: POST '/api/0/internal/mail/'>)

Could not get any other log on other pods. I think it is possiible at "SSL verfication" step. I coul not find any configuration parameter to set it "skip SSL sertification" to true.

mojitaleghani commented 3 weeks ago

I have the same error without using k8s.. I am running a self-hosted sentry and facing below errors:

02/Nov/2024:13:50:59 +0000] "POST /api/0/internal/mail/ HTTP/1.1" 500 42 "https://sentry.###/manage/status/mail/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"