search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.78k stars 1.76k forks source link

Sentry-self-hosted-worker-1 ignores the system root CAs #2950

Open lemrouch opened 5 months ago

lemrouch commented 5 months ago

Self-Hosted Version

24.2.0

CPU Architecture

x86_64

Docker Version

26.0.0

Docker Compose Version

2.20.3

Steps to Reproduce

Add our own CA to the certificates directory. Set system.url-prefix to https://sentry.ourdomain.com. Run self-hosted Sentry with certificate signed with our own CA.

I was able to work around the problem by appending our CA to the certifi module storage in the worker container. But I guess the worker container should use the system.internal-url-prefix instead.

Expected Result

No errors reported.

Actual Result

Logs from sentry-self-hosted-worker-1 contain:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1097, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 783, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 471, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 515, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 492, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/transport.py", line 535, in send_envelope_wrapper
    self._send_envelope(envelope)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/transport.py", line 434, in _send_envelope
    self._send_request(
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/sdk.py", line 249, in patched_send_request
    return _send_request(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/transport.py", line 245, in _send_request
    response = self._pool.request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/_request_methods.py", line 118, in request
    return self.request_encode_body(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/_request_methods.py", line 217, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 875, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 875, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 875, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sentry.ourdomain.com', port=443): Max retries exceeded with url: /api/1/envelope/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
00:54:55 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

Event ID

No response

azaslavsky commented 5 months ago

I wonder if this is https://github.com/getsentry/self-hosted/issues/2305 making a re-appearance...

lcsvcn commented 5 months ago

same error here at 24.4.2

roock commented 2 months ago

same issue with 24.5.1

my best guess would be that the sentry sdk inside the worker container/process does not honor the system certificate installed while starting the container. maybe the corresponding option in the sentry sdk needs to be set. (https://docs.sentry.io/platforms/python/configuration/options/#ca-certs)

lemrouch commented 1 month ago

Container sentry-self-hosted-sentry-cleanup-1 has the same problem. Workaround:

docker exec -it sentry-self-hosted-sentry-cleanup-1 bash
cat >> /.venv/lib/python3.11/site-packages/certifi/cacert.pem <<EOF
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
EOF

Do it in both containers. I don't think Certifi is willing to accept some environment variable to override it's built-in default.