search

sentry-kubernetes / charts

Easily deploy Sentry on your Kubernetes Cluster
MIT License
1.09k stars 516 forks source link

Bug Report: CSRF Verification Failed in Sentry Deployment Behind Treafik #1365

Closed voidsp-rick closed 1 week ago

voidsp-rick commented 3 months ago

Description

I'm encountering a persistent issue with CSRF verification in my Sentry deployment using the Helm chart. Despite configuring CSRF_TRUSTED_ORIGINS and other relevant settings, I continue to receive CSRF verification errors.

Environment

Configuration Details

Argo CD Application YAML
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ''
  namespace: argo-cd
spec:
  destination:
    namespace: sentry-system
    server: https://kubernetes.default.svc
  project: example-project
  source:
    chart: sentry
    helm:
      parameters:
        - name: sentry.singleOrganization
          value: 'false'
      valueFiles:
        - values.yaml
      values: |-
        user:
          create: true
          email: admin@example.com
          password: ExamplePassword123

        # Database settings
        postgresql:
          existingSecret: "sentry-postgresql-secret"
          existingSecretKey: "postgresql-password"
          auth:
            existingSecret: sentry-postgresql-secret
            secretKeys:
              adminPasswordKey: postgresql-password

        ingress:
          enabled: true
          regexPathStyle: traefik
          annotations:
            kubernetes.io/ingress.class: traefik
            traefik.ingress.kubernetes.io/router.middlewares: kube-system-default-https-redirect@kubernetescrd,kube-system-default-x-forwarded-proto@kubernetescrd
          hostname: sentry.example.com
          tls:
          - secretName: sentry.example.com-tls

        system:
          url: "https://sentry.example.com"
          adminEmail: "admin@example.com"
          public: false

        mail:
          backend: smtp
          useTls: false
          useSsl: false
          username: system@example.com
          password: ExamplePassword123
          port: 2525
          host: smtp.example.com
          from: system@example.com

        google:
          existingSecret: "sentry-google-secret"
          existingSecretClientIdKey: "client-id"
          existingSecretClientSecretKey: "client-secret"

        nginx:
          enabled: false

        config:
          sentryConfPy: |
            CSRF_TRUSTED_ORIGINS = ["https://sentry.example.com"]
            SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
            SESSION_COOKIE_SECURE = True
            CSRF_COOKIE_SECURE = True
            SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

        hooks:
          dbInit:
            enabled: false
    repoURL: https://sentry-kubernetes.github.io/charts
    targetRevision: 23.12.0
Ingress Configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-sentry-system
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.middlewares: kube-system-default-https-redirect@kubernetescrd,kube-system-default-x-forwarded-proto@kubernetescrd
  labels:
    app: example-sentry-system
    argocd.argoproj.io/instance: example-sentry-system
    chart: sentry-23.12.0
    heritage: Helm
    release: example-sentry-system
  namespace: sentry-system
spec:
  rules:
    - host: sentry.example.com
      http:
        paths:
          - backend:
              service:
                name: example-sentry-system-relay
                port:
                  number: 3000
            path: /api/store
            pathType: ImplementationSpecific
          - backend:
              service:
                name: example-sentry-system-relay
                port:
                  number: 3000
            path: /api/{[1-9][0-9]*}/{(.*)}
            pathType: ImplementationSpecific
          - backend:
              service:
                name: example-sentry-system-web
                port:
                  number: 9000
            path: /
            pathType: ImplementationSpecific
  tls:
    - secretName: sentry.example.com-tls

Issue Details

  1. CSRF Verification Errors:
    • Despite setting CSRF_TRUSTED_ORIGINS and related configurations, I am consistently receiving CSRF verification errors.
    • Error message: 403 Forbidden (Referer checking failed - no Referer.)

Steps Taken

  1. Configuration Verification:

    • Verified configurations via Django shell: 
      from django.conf import settings
print(settings.CSRF_TRUSTED_ORIGINS)
print(settings.SECURE_PROXY_SSL_HEADER)
print(settings.SESSION_COOKIE_SECURE)
print(settings.CSRF_COOKIE_SECURE)
print(settings.SOCIAL_AUTH_REDIRECT_IS_HTTPS)
    • Output matches the expected settings: 
      ['https://sentry.example.com']
('HTTP_X_FORWARDED_PROTO', 'https')
True
True
True

  2. Ingress Configuration:

    • Verified that Ingress annotations are set correctly to handle SSL redirection and X-Forwarded-Proto headers.

  3. Pod Restarts:

    • Restarted Sentry web, relay, and worker pods to ensure configurations are reloaded.

Error Logs

Sentry Web Log:

*** Starting uWSGI 2.0.23 (64bit) on [Sat Jul 13 10:44:40 2024] ***
compiled with version: 10.2.1 20210130 (Red Hat 10.2.1-11) on 10 January 2024 21:25:28
os: Linux-5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024
nodename: example-sentry-system-web-77bd558cd6-2k7wf
machine: x86_64
...
10:44:49 [WARNING] django.security.csrf: Forbidden (Referer checking failed - no Referer.): /api/2/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/2/envelope/'>)
10:44:49 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/2/envelope/' caller_ip='192.168.50.5' user_agent='sentry.php.wordpress/7.20.0' rate_limited='False' rate_limit_category='None' 
...

Sentry Worker Log:

10:44:15 [INFO] sentry.bgtasks: bgtask.spawn (task_name='sentry.bgtasks.clean_dsymcache:clean_dsymcache')
10:44:15 [INFO] sentry.bgtasks: bgtask.spawn (task_name='sentry.bgtasks.clean_releasefilecache:clean_releasefilecache')
/.venv/lib/python3.11/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
  warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(

10:45:00 [ERROR] sentry_sdk.errors: Unexpected status code: 403 (body: b'\n\n\n\n\n\n\n\n\n\n<!DOCTYPE html>\n<html lang="en">\n<head>\n  <meta http-equiv="content-type" content="text/html; charset=utf-8">\n  <!-- The "none" directive is equivalent to using both the noindex and nofollow tags simultaneously -->\n  <meta name="robots" content="none, noarchive">\n  <meta name="viewport" content="width=device-width, initial-scale=1">\n\n  <link rel="icon" type="image/png" href="https://sentry.example.com/_static/1720867481/sentry/images/favicon.png">\n\n  <link rel="apple-touch-icon" href="https://sentry.example.com/_static/1720867481/sentry/images/logos/apple-touch-icon.png">\n  <link rel="apple-touch-icon" sizes="76x76" href="https://sentry.example.com/_static/1720867481/sentry/images/logos/apple-touch-icon-76x76.png">\n  <link rel="apple-touch-icon" sizes="120x120" href="https://sentry.example.com/_static/1720867481/sentry/images/logos/apple-touch-icon-120x120.png">\n  <link rel="apple-touch-icon" sizes="152x152" href="https://sentry.example.com/_static/1720867481/sentry/images/logos/apple-touch-icon-152x152.png">\n  \n\n  <link rel="mask-icon

My domain is served behind cloud flare with proxy full mode. UI is accessible but with the same error.

➜  ~ curl --data '{a:1}' -H 'Content-Type: application/json' -H "X-Sentry-Auth: Sentry sentry_version=7, sentry_key=example_key, sentry_client=raven-bash/0.1" https://sentry.example.com/api/2/store/

Response:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8">
  <!-- The "none" directive is equivalent to using both the noindex and nofollow tags simultaneously -->
  <meta name="robots" content="none, noarchive">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <link rel="icon" type="image/png" href="https://sentry.example.com/_static/1720868364/sentry/images/favicon.png">

  <link rel="apple-touch-icon" href="https://sentry.example.com/_static/1720868364/sentry/images/logos/apple-touch-icon.png">
  <link rel="apple-touch-icon" sizes="76x76" href="https://sentry.example.com/_static/1720868364/sentry/images/logos/apple-touch-icon-76x76.png">
  <link rel="apple-touch-icon" sizes="120x120" href="https://sentry.example.com/_static/1720868364/sentry/images/logos/apple-touch-icon-120x120.png">
  <link rel="apple-touch-icon" sizes="152x152" href="https://sentry.example.com/_static/1720868364/sentry/images/logos/apple-touch-icon-152x152.png">

  <link rel="mask-icon" sizes="any" href="https://sentry.example.com/_static/1720868364/sentry/images/logos/logo-sentry.svg" color="#FB4226">

  <link href="/_static/dist/sentry/entrypoints/sentry.css" rel="stylesheet"/>

  <title>CSRF Verification Failed | Sentry</title>

    <script nonce="PQe4i/cNshyR7vAiv2NEWg==">window.__initialData = {"initialTrace":{"sentry_trace":"eeecf7939b67424c9f074121d002d6b6-b6d27ec8a6a408c4-0","baggage":"sentry-trace_id=eeecf7939b67424c9f074121d002d6b6,sentry-environment=production,sentry-release=backend%4024.5.1%2Bunknown,sentry-transaction=/api/%5B0%5D%2B/,sentry-sample_rate=0.0,sentry-sampled=false"},"customerDomain":null,"singleOrganization":false,"supportEmail":"admin@example.com","urlPrefix":"https://sentry.example.com","version":{"current":"24.5.1","latest":"24.6.0","build":"unknown","upgradeAvailable":true},"features":["organizations:create","organizations:multi-region-selector"],"distPrefix":"/_static/dist/sentry/","needsUpgrade":false,"dsn":"https://example_dsn@sentry.example.com/1","statuspage":null,"messages":[],"apmSampling":0.0,"isOnPremise":true,"isSelfHosted":true,"isSelfHostedErrorsOnly":false,"shouldPreloadData":false,"shouldShowBeaconConsentPrompt":false,"invitesEnabled":true,"gravatarBaseUrl":"https://gravatar.com","termsUrl":null,"privacyUrl":null,"lastOrganization":null,"languageCode":"en","userIdentity":{"ip_address":"192.168.50.5"},"csrfCookieName":"sc","superUserCookieName":"su","superUserCookieDomain":null,"sentryConfig":{"dsn":"https://example_dsn@sentry.example.com/1","release":"frontend@24.5.1+unknown","environment":"production","whitelistUrls":[],"allowUrls":[],"tracePropagationTargets":[]},"memberRegions":[],"regions":[{"name":"--monolith--","url":"https://sentry.example.com"}],"relocationConfig":{"selectableRegions":[]},"demoMode":false,"enableAnalytics":false,"validateSUForm":true,"disableU2FForSUForm":false,"links":{"organizationUrl":null,"regionUrl":null,"sentryUrl":"https://sentry.example.com"},"user":null,"isAuthenticated":false};</script>

  <script nonce="PQe4i/cNshyR7vAiv2NEWg==">// if the ads.js file loads below it will mark this variable as false
    window.adblockSuspected = true;
    // Initialize this so that we can queue up tasks when Sentry SPA is initialized
    window.__onSentryInit = window.__onSentryInit || [];</script>

<meta name="sentry-trace" content="eeecf7939b67424c9f074121d002d6b6-b6d27ec8a6a408c4-0">
<meta name="baggage" content="sentry-trace_id=eeecf7939b67424c9f074121d002d6b6,sentry-environment=production,sentry-release=backend%4024.5.1%2Bunknown,sentry-transaction=/api/%5B0%5D%2B/,sentry-sample_rate=0.0,sentry-sampled=false">

<script nonce="PQe4i/cNshyR7vAiv2NEWg==">function __preloadData() {
    if (!window.__initialData.shouldPreloadData) {
      return;
    }
    var slug = window.__initialData.lastOrganization;
    if (!slug && window.__initialData.customerDomain) {
      slug = window.__initialData.customerDomain.subdomain;
    }
    var host = '';
    if (window.__initialData.links && window.__initialData.links.regionUrl !== window.__initialData.links.sentryUrl) {
      var host = window.__initialData.links.regionUrl;
    }

    function promiseRequest(url) {
      return new Promise(function (resolve, reject) {
        var xhr = new XMLHttpRequest();
        xhr.open('GET', url);
        xhr.setRequestHeader("sentry-trace", window.__initialData.initialTrace.sentry_trace);
        xhr.setRequestHeader("baggage", window.__initialData.initialTrace.baggage);
        xhr.withCredentials = true;
        xhr.onload = function () {
          try {
            this.status >= 200 && this.status < 300
              ? resolve([JSON.parse(xhr.response), this.statusText, xhr])
              : reject([this.status, this.statusText]);
          } catch (e) {
            reject();
          }
        };
        xhr.onerror = function () {
          reject([this.status, this.statusText]);
        };
        xhr.send();
      });
    }

    function makeUrl(suffix) {
      return host + '/api/0/organizations/' + slug + suffix;
    }

    var preloadPromises = {orgSlug: slug};
    window.__sentry_preload = preloadPromises;

    if (!slug) {
      return;
    }

    preloadPromises.organization = promiseRequest(makeUrl('/?detailed=0&include_feature_flags=1'));
    preloadPromises.projects = promiseRequest(
      makeUrl('/projects/?all_projects=1&collapse=latestDeploys&collapse=unusedFeatures')
    );
    preloadPromises.teams = promiseRequest(makeUrl('/teams/'));
  }

  try {
    __preloadData();
  } catch (_) {}</script>

    <script crossorigin="anonymous" nonce="PQe4i/cNshyR7vAiv2NEWg==" src="/_static/dist/sentry/entrypoints/app.js"></script>
  <script crossorigin="anonymous" nonce="PQe4i/cNshyR7vAiv2NEWg==" src="/_static/1720868364/sentry/js/ads.js"></script>

</head>
<body class=" narrow">
  <div class="app">
<div id="blk_alerts" class="messages-container"></div>
<div id="blk_indicators"></div>
<script nonce="PQe4i/cNshyR7vAiv2NEWg==">window.__onSentryInit = window.__onSentryInit || [];
  window.__onSentryInit.push({
    name: 'renderReact',
    component: 'SystemAlerts',
    container: '#blk_alerts',
    props: {
      className: 'alert-list',
    },
  });
  window.__onSentryInit.push({
    name: 'renderReact ',
    component: 'Indicators',
    container: '#blk_indicators',
    props: {
      className: 'indicators-container',
    },
  });</script>

    <div class="container">
      <div class="content">
<div class="pattern-bg"></div>
<section class="org-login">
  <div class="box box-modal">
    <div class="box-header">
        <a class="logo-with-action" href="/">
            <span class="icon-sentry-logo-full"></span>
        </a>
    </div>
    <div class="box-content with-padding">
    <section class="body">
        <div class="page-header">
            <h2>CSRF Verification Failed</h2>
        </div>

        <p>A required security token was not found or was invalid.</p>

        <p>If you're continually seeing this issue, try the following:</p>

        <ol>
lbcd commented 3 months ago

Same thing for me after a fresh installation. Do you find a solution?

KoCoder commented 3 months ago

When I made requests using the Sentry java client, I found out, that sentry relay doesn't log any requests.

I don't know why Traefik doesn't route traffic to sentry-relay, I will debugging this issue further.

A simple workarround I found was to issue a new Subdomain (for example sentry2.sentry.local) and configure it as the sentry dsn in your clients. You'd have to write your own ingress and add this domain to the CSRF Trusted Origin setting.

This is the ingressroute that works for me

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: sentry-ingressroute
  namespace: sentry
  annotations:
      kubernetes.io/ingress.class: traefik
spec:
  entryPoints:
    - websecure
  routes:
  - kind: Rule
    match: Host(`sentry2.redacted.tld`)
    services:
    - kind: Service
      name: sentry-relay
      port: 3000
      namespace: sentry
  tls: {}
zvickery commented 2 months ago

One cause of this is that Traefik v3 by default does not respect regular expressions in k8s Ingress objects. I was able to fix this by setting --core.defaultRuleSyntax=v2 in Traefik and then things started working as expected.

The traefik.ingress.kubernetes.io/router.pathmatcher and traefik.ingress.kubernetes.io/router.rulesyntax (the latter not yet shipped) ingress annotations may also help here. I intend to investigate these once the next version of Traefik is released.

tyr1k commented 1 month ago

@zvickery thanks a lot --core.defaultRuleSyntax=v2 fix same problem

prasadkris commented 1 month ago

@zvickery - Thanks a lot for sharing the workaround! I had the exact same issue after upgrading to Traefik v3, and switching to the v2 rule syntax helped fix it! 🙏🏻

zvickery commented 1 month ago

With Traefik v3.1.3 I have confirmed that the below Ingress annotation also works. This allows setting v2 style path matching only for this Ingress and not globally:

  annotations:
    traefik.ingress.kubernetes.io/router.rulesyntax: v2
Mokto commented 3 weeks ago

This issue is stale because it has been open for 30 days with no activity.

Mokto commented 1 week ago

This issue was closed because it has been inactive for 14 days since being marked as stale.