search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.96k stars 1.78k forks source link

Event submission rejected by django CSRF: Forbidden (Referer checking failed - no Referer.) #3312

Closed razvan286 closed 1 week ago

razvan286 commented 2 months ago

Self-Hosted Version

24.8.0

CPU Architecture

x86_64

Docker Version

27.2.0

Docker Compose Version

2.29.2

Steps to Reproduce

Clone the self-hosted Sentry repository.

Sentry configuration:

cat sentry/config.yml:

###############
# Mail Server #
###############

# mail.backend: 'smtp'  # Use dummy if you want to disable email entirely
mail.host: 'smtp'

###################
# System Settings #
###################

# If this file ever becomes compromised, it's important to generate a new key.
# Changing this value will result in all current sessions being invalidated.
# A new key can be generated with `$ sentry config generate-secret-key`
system.secret-key: '*******'

################
# File storage #
################

# Uploaded media uses these `filestore` settings. The available
# backends are either `filesystem` or `s3`.

filestore.backend: 'filesystem'
filestore.options:
  location: '/data/files'
dsym.cache-path: '/data/dsym-cache'
releasefile.cache-path: '/data/releasefile-cache'

# The URL prefix in which Sentry is accessible
system.internal-url-prefix: 'http://web:9000'
symbolicator.enabled: true
symbolicator.options:
  url: "http://symbolicator:3021"

transaction-events.force-disable-internal-project: true

system.url-prefix: 'https://sentry.mydomain.com'

mail.host: 'smtp.mail.com'
mail.port: 587
mail.username: 'noreply@mail.com'
mail.password: '****'

cat sentry/sentry.conf.py:

# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.

from sentry.conf.server import *  # NOQA

BYTE_MULTIPLIER = 1024
UNITS = ("K", "M", "G")

def unit_text_to_bytes(text):
    unit = text[-1].upper()
    power = UNITS.index(unit) + 1
    return float(text[:-1]) * (BYTE_MULTIPLIER**power)

# Generously adapted from pynetlinux: https://github.com/rlisagor/pynetlinux/blob/e3f16978855c6649685f0c43d4c3fcf768427ae5/pynetlinux/ifconfig.py#L197-L223
def get_internal_network():
    import ctypes
    import fcntl
    import math
    import socket
    import struct

    iface = b"eth0"
    sockfd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    ifreq = struct.pack(b"16sH14s", iface, socket.AF_INET, b"\x00" * 14)

    try:
        ip = struct.unpack(
            b"!I", struct.unpack(b"16sH2x4s8x", fcntl.ioctl(sockfd, 0x8915, ifreq))[2]
        )[0]
        netmask = socket.ntohl(
            struct.unpack(b"16sH2xI8x", fcntl.ioctl(sockfd, 0x891B, ifreq))[2]
        )
    except IOError:
        return ()
    base = socket.inet_ntoa(struct.pack(b"!I", ip & netmask))
    netmask_bits = 32 - int(round(math.log(ctypes.c_uint32(~netmask).value + 1, 2), 1))
    return "{0:s}/{1:d}".format(base, netmask_bits)

INTERNAL_SYSTEM_IPS = (get_internal_network(),)

DATABASES = {
    "default": {
        "ENGINE": "sentry.db.postgres",
        "NAME": "postgres",
        "USER": "postgres",
        "PASSWORD": "",
        "HOST": "postgres",
        "PORT": "",
    }
}

# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True

# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings

###########
# General #
###########

# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True

SENTRY_OPTIONS["system.event-retention-days"] = int(
    env("SENTRY_EVENT_RETENTION_DAYS", "90")
)

#########
# Redis #
#########

# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB

SENTRY_OPTIONS["redis.clusters"] = {
    "default": {
        "hosts": {0: {"host": "redis", "password": "", "port": "6379", "db": "0"}}
    }
}

#########
# Queue #
#########

# See https://develop.sentry.dev/services/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.

rabbitmq_host = None
if rabbitmq_host:
    BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
        username="guest", password="guest", host=rabbitmq_host, vhost="/"
    )
else:
    BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
        **SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
    )

#########
# Cache #
#########

# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache",
        "LOCATION": ["memcached:11211"],
        "TIMEOUT": 3600,
        "OPTIONS": {"ignore_exc": True},
    }
}

# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"

DEFAULT_KAFKA_OPTIONS = {
    "bootstrap.servers": "kafka:9092",
    "message.max.bytes": 50000000,
    "socket.timeout.ms": 1000,
}

SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream"
SENTRY_EVENTSTREAM_OPTIONS = {"producer_configuration": DEFAULT_KAFKA_OPTIONS}

KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS

###############
# Rate Limits #
###############

# Rate limits apply to notification handlers and are enforced per-project
# automatically.

SENTRY_RATELIMITER = "sentry.ratelimits.redis.RedisRateLimiter"

##################
# Update Buffers #
##################

# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)

SENTRY_BUFFER = "sentry.buffer.redis.RedisBuffer"

##########
# Quotas #
##########

# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.

SENTRY_QUOTAS = "sentry.quotas.redis.RedisQuota"

########
# TSDB #
########

# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.

SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"

#########
# SNUBA #
#########

SENTRY_SEARCH = "sentry.search.snuba.EventsDatasetSnubaSearchBackend"
SENTRY_SEARCH_OPTIONS = {}
SENTRY_TAGSTORE_OPTIONS = {}

###########
# Digests #
###########

# The digest backend powers notification summaries.

SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"

###################
# Metrics Backend #
###################

SENTRY_RELEASE_HEALTH = "sentry.release_health.metrics.MetricsReleaseHealthBackend"
SENTRY_RELEASE_MONITOR = (
    "sentry.release_health.release_monitor.metrics.MetricReleaseMonitorBackend"
)

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed in order to prevent https://github.com/getsentry/sentry/blob/c6f9660e37fcd9c1bbda8ff4af1dcfd0442f5155/src/sentry/services/http.py#L70
    "uwsgi-socket": None,
    "so-keepalive": True,
    # Keep this between 15s-75s as that's what Relay supports
    "http-keepalive": 15,
    "http-chunked-input": True,
    # the number of web workers
    "workers": 3,
    "threads": 4,
    "memory-report": False,
    # Some stuff so uwsgi will cycle workers sensibly
    "max-requests": 100000,
    "max-requests-delta": 500,
    "max-worker-lifetime": 86400,
    # Duplicate options from sentry default just so we don't get
    # bit by sentry changing a default value that we depend on.
    "thunder-lock": True,
    "log-x-forwarded-for": False,
    "buffer-size": 32768,
    "limit-post": 209715200,
    "disable-logging": True,
    "reload-on-rss": 600,
    "ignore-sigpipe": True,
    "ignore-write-errors": True,
    "disable-write-exception": True,
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
USE_X_FORWARDED_HOST = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

# End of SSL/TLS settings

########
# Mail #
########

SENTRY_OPTIONS["mail.list-namespace"] = env("SENTRY_MAIL_HOST", "localhost")
SENTRY_OPTIONS["mail.from"] = f"sentry@{SENTRY_OPTIONS['mail.list-namespace']}"

############
# Features #
############

SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            "organizations:discover",
            "organizations:events",
            "organizations:global-views",
            "organizations:incidents",
            "organizations:integrations-issue-basic",
            "organizations:integrations-issue-sync",
            "organizations:invite-members",
            "organizations:metric-alert-builder-aggregate",
            "organizations:sso-basic",
            "organizations:sso-rippling",
            "organizations:sso-saml2",
            "organizations:performance-view",
            "organizations:advanced-search",
            "organizations:session-replay",
            "organizations:issue-platform",
            "organizations:profiling",
            "organizations:monitors",
            "organizations:dashboards-mep",
            "organizations:mep-rollout-flag",
            "organizations:dashboards-rh-widget",
            "organizations:metrics-extraction",
            "organizations:transaction-metrics-extraction",
            "projects:custom-inbound-filters",
            "projects:data-forwarding",
            "projects:discard-groups",
            "projects:plugins",
            "projects:rate-limits",
            "projects:servicehooks",
        )
        # Starfish related flags
        + (
            "organizations:deprecate-fid-from-performance-score",
            "organizations:indexed-spans-extraction",
            "organizations:insights-entry-points",
            "organizations:insights-initial-modules",
            "organizations:insights-addon-modules",
            "organizations:mobile-ttid-ttfd-contribution",
            "organizations:performance-calculate-score-relay",
            "organizations:standalone-span-ingestion",
            "organizations:starfish-browser-resource-module-image-view",
            "organizations:starfish-browser-resource-module-ui",
            "organizations:starfish-browser-webvitals",
            "organizations:starfish-browser-webvitals-pageoverview-v2",
            "organizations:starfish-browser-webvitals-replace-fid-with-inp",
            "organizations:starfish-browser-webvitals-use-backend-scores",
            "organizations:starfish-mobile-appstart",
            "projects:span-metrics-extraction",
            "projects:span-metrics-extraction-addons",
        )
        # User Feedback related flags
        + (
            "organizations:user-feedback-ingest",
            "organizations:user-feedback-replay-clip",
            "organizations:user-feedback-ui",
            "organizations:feedback-visible",
            "organizations:feedback-ingest",
            "organizations:feedback-post-process-group",
        )
    }
)

# Temporary flag to mark User Feedback to be produced to the dedicated feedback topic by relay.
# This will be removed at a later time after it's considered stable and fully rolled out.
SENTRY_OPTIONS["feedback.ingest-topic.rollout-rate"] = 1.0

#######################
# MaxMind Integration #
#######################

GEOIP_PATH_MMDB = "/geoip/GeoLite2-City.mmdb"

##############################################
# Suggested Fix Feature / OpenAI Integration #
##############################################

# See https://docs.sentry.io/product/issues/issue-details/ai-suggested-solution/
# for more information about the feature. Make sure the OpenAI's privacy policy is
# aligned with your company.

# Set the OPENAI_API_KEY on the .env or .env.custom file with a valid
# OpenAI API key to turn on the feature.
OPENAI_API_KEY = env("OPENAI_API_KEY", "")

SENTRY_FEATURES["organizations:open-ai-suggestion"] = bool(OPENAI_API_KEY)

##############################################
# Content Security Policy settings
##############################################

# CSP_REPORT_URI = "https://{your-sentry-installation}/api/{csp-project}/security/?sentry_key={sentry-key}"
CSP_REPORT_ONLY = True

# optional extra permissions
# https://django-csp.readthedocs.io/en/latest/configuration.html
# CSP_SCRIPT_SRC += ["example.com"]

#################
# CSRF Settings #
#################

# Since version 24.1.0, Sentry migrated to Django 4 which contains stricter CSRF protection.
# If you are accessing Sentry from multiple domains behind a reverse proxy, you should set
# this to match your IPs/domains. Ports should be included if you are using custom ports.
# https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-CSRF_TRUSTED_ORIGINS

CSRF_TRUSTED_ORIGINS = ["*", "https://*sentry.mydomain.com", "https://*", "http://*"]

#################
# JS SDK Loader #
#################

JS_SDK_LOADER_DEFAULT_SDK_URL = "https://browser.sentry-cdn.com/%s/bundle%s.min.js"

# If you would like to use self-hosted Sentry with only errors enabled, please set this
SENTRY_SELF_HOSTED_ERRORS_ONLY = env("COMPOSE_PROFILES") != "feature-complete"

Caddy configuration

I use Caddy for setting up the reverse proxy by adding the caddy container in a docker-compose.override.yml file. cat docker-compose.override.yml:

services:
  caddy:
    image: caddy:2.8.4
    container_name: caddy
    restart: always
    ports:
      - 80:80  # Needed for the ACME HTTP-01 challenge.
      - 443:443
      - 443:443/udp # Needed for HTTP/3.
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data
    environment:
      DOMAIN: "sentry.mydomain.com"
      EMAIL: "email@mail.com"  # The email address to use for ACME registration.
      LOG_FILE: "/data/access.log"

This is the Caddyfile available in the root directory of the cloned repo.

cat Caddyfile:

{
    debug
}

sentry.mydomain.com {

    log {
        output file /data/access.log
        format json
    }
    reverse_proxy web:9000 {
        health_uri /_health/
        health_status 2xx
        header_up Host {upstream_hostport}
    }

    # By default, the TLS is acquired from Let's Encrypt
    tls email@mail.com

    # If you have self-signed certificate
    # tls /path/to/server-certificate.crt /path/to/server-certificate.key

    # To enable rate limiter, install additional module from
    # https://github.com/mholt/caddy-ratelimit
    # rate_limit {
    #     zone sentry {
    #         key {remote_host}
    #         window 1s
    #         events 100
    #     }
    # }

    # To expose only ingest endpoint publicly, add the named matcher below before `reverse_proxy` directive
    # @ingest_endpoint {
    #     path_regexp /api/[1-9]\d+/(envelope|minidump|security|store|unreal)/
    # }
}

Run ./install.sh and then spin up docker container docker compose up. All containers running then set up a Sentry python project on sentry.mydomain.com. Get the DSN and create the python script:

Test connection using a simple python script:

cat test-self-hosted-sentry.py:

import sentry_sdk

sentry_sdk.init(
    dsn="https://*****@sentry.mydomain.com/3",
    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for tracing.
    traces_sample_rate=1.0,
)

division_by_zero = 1 / 0

Expected Result

A new issue is reported under the Sentry python project at URL: https://sentry.mydomain.com/organizations/sentry/projects/python/?project=3.

Actual Result

docker compose logs -f web:

web-1  | 11:07:26 [WARNING] django.security.csrf: Forbidden (Referer checking failed - no Referer.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1  | 11:07:26 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='*****' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.018622398376464844 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

Event ID

No response

aldy505 commented 2 months ago

On your Caddyfile, the reverse_proxy directive should point to nginx, which binds to port 80. https://github.com/getsentry/self-hosted/blob/b8b4aa20aa7aa4e37a8a2569d70ca54d450ae947/docker-compose.yml#L455

That should probably fix it. But I see your sentry.conf.py having this:

CSRF_TRUSTED_ORIGINS = ["*", "https://*sentry.mydomain.com", "https://*", "http://*"]

I'm not sure whether Django will accept that. If I were you, I'll just do this:

CSRF_TRUSTED_ORIGINS = ["http://sentry.mydomain.com", "https://sentry.mydomain.com"]
razvan286 commented 2 months ago

Thank you for your response. I have updated the Caddyfile based on your suggestion, and also the CSRF_TRUSTED_ORIGINS value. cat Caddyfile:

{
    debug
}

{$DOMAIN} {

    log {
        output file /data/access.log
        format json
    }

    reverse_proxy nginx:80 {
        health_uri /_health/
        health_status 2xx
        header_up Host {upstream_hostport}
    }
    # By default, the TLS is acquired from Let's Encrypt
    tls email@mail.com
}

By checking the docker containers logs it seems that the requests now go through caddy and also nginx and relay until they reach the sentry instance (web-1). However, now there is a different error reported by django:

web-1                                           | 09:30:29 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1                                           | 09:30:29 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='****' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016865015029907227 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
aldy505 commented 2 months ago

By checking the docker containers logs it seems that the requests now go through caddy and also nginx and relay until they reach the sentry instance (web-1). However, now there is a different error reported by django:

web-1 | 09:30:29 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>) web-1 | 09:30:29 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='****' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016865015029907227 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

@razvan286 after changing your sentry.conf.py file, most of the time you'll need to re-run ./install.sh script.

razvan286 commented 2 months ago

@aldy505 yes, I always run the ./install.sh and then the docker compose up command.

aldy505 commented 2 months ago

Oh I just realized this message

09:30:29 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)

The /api/3/envelope should be pointed to the relay container instead of the web container.

Can you confirm from your Caddy instance that it really did a reverse proxy to the nginx container, and then correctly did a reverse proxy to the relay container?

To sum up (the name of the container):

caddy -> nginx -> relay

Then data ingested by relay should be published to kafka as individual messages, then it would be consumed by one of sentry worker (in which this becomes more complicated)

razvan286 commented 2 months ago

@aldy505 Here is the full container output from when the request reaches caddy first and in the end the sentry instance that returns the CSRF missing token error:

caddy                                           | {"level":"debug","ts":1725955821.389444,"logger":"events","msg":"event","name":"tls_get_certificate","id":"fcf0f3b8-9a92-4ca3-8ae7-94fa1e16f8c7","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,49195,49199,52393,52392,49188,49192,49187,49191,159,158,107,103,255],"ServerName":"sentry.mydomain.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"****","Port":37110,"Zone":""},"LocalAddr":{"IP":"*****","Port":443,"Zone":""}}}}
caddy                                           | {"level":"debug","ts":1725955821.3895462,"logger":"tls.handshake","msg":"choosing certificate","identifier":"sentry.mydomain.com","num_choices":1}
caddy                                           | {"level":"debug","ts":1725955821.3895824,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"sentry.mydomain.com","subjects":["sentry.mydomain.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1725955821.389603,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"*****","remote_port":"37110","subjects":["sentry.mydomain.com"],"managed":true,"expiration":1733305272,"hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1725955821.4079857,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"nginx:80","total_upstreams":1}
relay-1                                         | 2024-09-10T08:10:21.409512Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_request: started processing request
relay-1                                         | 2024-09-10T08:10:21.410400Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-10T08:10:21.410441Z DEBUG relay_server::services::project: project 6c77219228f5973a3f9efc00978f35ae state requested 1 times
relay-1                                         | 2024-09-10T08:10:21.410558Z TRACE request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: relay_server::endpoints::common: Sending envelope to project cache for V1 buffer
relay-1                                         | 2024-09-10T08:10:21.410748Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=1 ms status=200
relay-1                                         | 2024-09-10T08:10:21.411020Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-10T08:10:21.411044Z TRACE relay_server::services::project_cache: Enqueueing envelope
nginx-1                                         | *** - - [10/Sep/2024:08:10:21 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.13.0" "***"
caddy                                           | {"level":"debug","ts":1725955821.41155,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"nginx:80","duration":0.003337724,"request":{"remote_ip":"***","remote_port":"37110","client_ip":"***","proto":"HTTP/1.1","method":"POST","host":"nginx:80","uri":"/api/3/envelope/","headers":{"Content-Type":["application/x-sentry-envelope"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["identity"],"Content-Length":["1079"],"X-Forwarded-Host":["sentry.mydomain.com"],"Content-Encoding":["gzip"],"User-Agent":["sentry.python/2.13.0"],"X-Sentry-Auth":["Sentry sentry_key=6c77219228f5973a3f9efc00978f35ae, sentry_version=7, sentry_client=sentry.python/2.13.0"],"X-Forwarded-For":["***"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"sentry.mydomain.com"}},"headers":{"Connection":["keep-alive"],"Access-Control-Allow-Origin":["*"],"Vary":["origin","access-control-request-method","access-control-request-headers"],"Cross-Origin-Resource-Policy":["cross-origin"],"Server":["nginx"],"Date":["Tue, 10 Sep 2024 08:10:21 GMT"],"Content-Type":["application/json"],"Content-Length":["41"],"Access-Control-Expose-Headers":["x-sentry-error,x-sentry-rate-limits,retry-after"]},"status":200}
relay-1                                         | 2024-09-10T08:10:21.512620Z DEBUG relay_server::services::project_upstream: updating project states for 1/1 projects (attempt 1)
relay-1                                         | 2024-09-10T08:10:21.512811Z DEBUG relay_server::services::project_upstream: sending request of size 1
relay-1                                         | 2024-09-10T08:10:21.539993Z DEBUG relay_server::services::project: project state 6c77219228f5973a3f9efc00978f35ae updated
nginx-1                                         | 172.18.0.57 - - [10/Sep/2024:08:10:27 +0000] "GET /_health/ HTTP/1.1" 200 12 "-" "Go-http-client/1.1" "-"
caddy                                           | {"level":"info","ts":1725955827.935051,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"host is up","host":"nginx:80"}
relay-1                                         | 2024-09-10T08:10:35.557499Z TRACE relay_server::services::metrics::aggregator: flushing 1 partitions to receiver
relay-1                                         | 2024-09-10T08:10:35.557858Z TRACE relay_server::services::processor: sending envelope to sentry endpoint
relay-1                                         | 2024-09-10T08:10:35.558493Z TRACE relay_server::services::cogs: recording measurement: CogsMeasurement { resource: Relay, feature: MetricsSessions, value: Time(724.815µs) }
web-1                                           | 08:10:35 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/1/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/1/envelope/'>)
web-1                                           | 08:10:35 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/1/envelope/' caller_ip='' user_agent='sentry-relay/24.8.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.03174233436584473 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
aldy505 commented 2 months ago

That looks fine to me, the error you're trying to send from the Python app went through the projectId of 3 (as seen on /api/3/envelope). Yet the web container is trying to make a request to /api/1/envelope, in which I guess there is a frontend or backend error happening from the web container, but it wrongly being sent to web container instead of the relay or nginx container.

I hope this illustration explain the situation better: Image

For now, my suggestion is to modify the docker-compose.yml on this section: https://github.com/getsentry/self-hosted/blob/b8b4aa20aa7aa4e37a8a2569d70ca54d450ae947/docker-compose.yml#L43-L62

to be:

  environment:
    # ...
    OPENAI_API_KEY:
    SENTRY_DSN:

Then on your .env or .env.custom, set the SENTRY_DSN to the DSN of the project named "internal" on your self-hosted instance. The URL to accessing this should be http://your-sentry-instance.com/settings/your-organization-name/projects/internal/keys/

Re-run ./install.sh and let's see what will happen next

razvan286 commented 2 months ago

I added the DSN value of the internal sentry project to the .env file. It looks like this: https://******@sentry.mydomain.com/1. Ran the ./install.sh and then docker compose up.

Logs right after I run the python script: docker compose logs -f:

caddy                                           | {"level":"debug","ts":1726043825.853992,"logger":"events","msg":"event","name":"tls_get_certificate","id":"466cbdef-1c4e-45b0-ad5e-c6df8e4e7705","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,49195,49199,52393,52392,49188,49192,49187,49191,159,158,107,103,255],"ServerName":"sentry.mydomain.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"****","Port":49580,"Zone":""},"LocalAddr":{"IP":"***","Port":443,"Zone":""}}}}
caddy                                           | {"level":"debug","ts":1726043825.8540905,"logger":"tls.handshake","msg":"choosing certificate","identifier":"sentry.mydomain.com","num_choices":1}
caddy                                           | {"level":"debug","ts":1726043825.854142,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"sentry.mydomain.com","subjects":["sentry.mydomain.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1726043825.8541822,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"****","remote_port":"49580","subjects":["sentry.mydomain.com"],"managed":true,"expiration":1733305272,"hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1726043825.8728714,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"nginx:80","total_upstreams":1}
relay-1                                         | 2024-09-11T08:37:05.874100Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_request: started processing request
relay-1                                         | 2024-09-11T08:37:05.875026Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875193Z TRACE request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: relay_server::endpoints::common: Sending envelope to project cache for V1 buffer
relay-1                                         | 2024-09-11T08:37:05.875284Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=1 ms status=200
relay-1                                         | 2024-09-11T08:37:05.875504Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875532Z TRACE relay_server::services::project_cache: Sending envelope to processor
nginx-1                                         | **** - - [11/Sep/2024:08:37:05 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.13.0" "****"
caddy                                           | {"level":"debug","ts":1726043825.875985,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"nginx:80","duration":0.002951075,"request":{"remote_ip":"*****","remote_port":"49580","client_ip":"*****","proto":"HTTP/1.1","method":"POST","host":"nginx:80","uri":"/api/3/envelope/","headers":{"X-Forwarded-Proto":["https"],"Accept-Encoding":["identity"],"X-Forwarded-For":["****"],"User-Agent":["sentry.python/2.13.0"],"Content-Encoding":["gzip"],"X-Sentry-Auth":["Sentry sentry_key=6c77219228f5973a3f9efc00978f35ae, sentry_version=7, sentry_client=sentry.python/2.13.0"],"Content-Length":["1079"],"Content-Type":["application/x-sentry-envelope"],"X-Forwarded-Host":["sentry.mydomain.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"sentry.mydomain.com"}},"headers":{"Date":["Wed, 11 Sep 2024 08:37:05 GMT"],"Content-Type":["application/json"],"Access-Control-Expose-Headers":["x-sentry-error,x-sentry-rate-limits,retry-after"],"Cross-Origin-Resource-Policy":["cross-origin"],"Server":["nginx"],"Content-Length":["41"],"Connection":["keep-alive"],"Access-Control-Allow-Origin":["*"],"Vary":["origin","access-control-request-method","access-control-request-headers"]},"status":200}
relay-1                                         | 2024-09-11T08:37:05.875548Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875738Z TRACE relay_server::services::processor: Processing error group
relay-1                                         | 2024-09-11T08:37:05.875780Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875803Z TRACE relay_server::services::processor::event: processing json event
relay-1                                         | 2024-09-11T08:37:05.876220Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.876267Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.876726Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.877681Z TRACE relay_server::services::processor: sending envelope to sentry endpoint
relay-1                                         | 2024-09-11T08:37:05.879065Z TRACE relay_server::services::cogs: recording measurement: CogsMeasurement { resource: Relay, feature: Errors, value: Time(3.160312ms) }
web-1                                           | 08:37:05 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1                                           | 08:37:05 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='172.18.0.56' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.058341026306152344 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

It looks like there is the same CSRF cookie not set error while the request is only made to /api/3/envelope now.

Maybe also this relay log is important?

relay-1                                         | 2024-09-11T08:37:05.875504Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
razvan286 commented 2 months ago

Thank you again for your time and ideas @aldy505 . However, I am still blocked on this issue, is there anything else that I can do or also provide you with more information so that maybe you can guide me further towards finding a solution?

aldy505 commented 2 months ago

All the logs and routes looks correct.

Maybe also this relay log is important?

relay-1 | 2024-09-11T08:37:05.875504Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point 1, expected a string

Yeah this might be it. Are you using an old SDK version? Or was this something that's not sent by you manually?

razvan286 commented 2 months ago

Thank you for your response @aldy505. The request is sent using the python script mentioned in the issue description and I was using:

"sentry-sdk": {
            "hashes": [
                "sha256:1e0e2eaf6dad918c7d1e0edac868a7bf20017b177f242cefe2a6bcd47955961d",
                "sha256:b8bc3dc51d06590df1291b7519b85c75e2ced4f28d9ea655b6d54033503b5bf4"
            ],
            "index": "pypi",
            "markers": "python_version >= '3.6'",
            "version": "==2.13.0"
        },

Now I also updated the SDK version to the latest 2.14.0 and I get the same error

aldy505 commented 2 months ago

Okay, sorry for taking a while responding this, been a busy week. I'm asking someone I know on the relay team on what might be causing that specific message.

aldy505 commented 1 month ago

Okay, got a clue. So you will need to export the sent envelope body payload for us to observe. You can send the envelope dump on this issue or on the Sentry's Discord.

For how to export/dump the envelope body payload on the Python SDK.. I don't really have a clue.

stayallive commented 1 month ago

I am not a 100% sure but enabling debug might print the envelope: https://docs.sentry.io/platforms/python/configuration/options/#debug. However not all SDKs do this so maybe Python also doesn't but a good and easy first step.

razvan286 commented 1 month ago

I set debug=True within the Python script.

[sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.anthropic.AnthropicIntegration: Anthropic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.ariadne.AriadneIntegration: ariadne is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.arq.ArqIntegration: Arq is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.asyncpg.AsyncPGIntegration: asyncpg not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.chalice.ChaliceIntegration: Chalice is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.clickhouse_driver.ClickhouseDriverIntegration: clickhouse-driver not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.cohere.CohereIntegration: Cohere not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.fastapi.FastApiIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.gql.GQLIntegration: gql is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.graphene.GrapheneIntegration: graphene is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.httpx.HttpxIntegration: httpx is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.huey.HueyIntegration: Huey is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.huggingface_hub.HuggingfaceHubIntegration: Huggingface not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.langchain.LangchainIntegration: langchain not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.loguru.LoguruIntegration: LOGURU is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.openai.OpenAIIntegration: OpenAI not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pymongo.PyMongoIntegration: Pymongo not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.quart.QuartIntegration: Quart is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlette.StarletteIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlite.StarliteIntegration: Starlite is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.strawberry.StrawberryIntegration: strawberry-graphql is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration boto3
 [sentry] DEBUG: Setting up previously not enabled integration flask
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
 [sentry] DEBUG: Setting up previously not enabled integration sqlalchemy
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Enabling integration boto3
 [sentry] DEBUG: Enabling integration flask
 [sentry] DEBUG: Enabling integration sqlalchemy
 [sentry] DEBUG: Setting SDK name to 'sentry.python.flask'
 [sentry] DEBUG: [Profiling] Setting up continuous profiler in thread mode
Traceback (most recent call last):
  File "/home/razvan/git/test-sentry-self-hosted/test-sentry.py", line 11, in <module>
    division_by_zero = 1 / 0
ZeroDivisionError: division by zero
 [sentry] DEBUG: atexit: got shutdown signal
 [sentry] DEBUG: atexit: shutting down client
 [sentry] DEBUG: Flushing HTTP transport
 [sentry] DEBUG: background worker got flush request
 [sentry] DEBUG: Sending envelope [envelope with 1 items (error)] project:3 host:sentry.mydomain.com
 [sentry] DEBUG: 2 event(s) pending on flush
Sentry is attempting to send 2 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit
 [sentry] DEBUG: background worker flushed
 [sentry] DEBUG: Killing HTTP transport
 [sentry] DEBUG: background worker got kill request

Logs received after running the .py script docker compose logs -f:

caddy                                           | {"level":"info","ts":1727856242.2462742,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"host is up","host":"nginx:80"}
caddy                                           | {"level":"debug","ts":1727856242.9146547,"logger":"events","msg":"event","name":"tls_get_certificate","id":"930c09f8-288e-4c4e-8c76-9eb91c63202a","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,49195,49199,52393,52392,49188,49192,49187,49191,159,158,107,103,255],"ServerName":"sentry.mydomain.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"****","Port":34752,"Zone":""},"LocalAddr":{"IP":"****","Port":443,"Zone":""}}}}
caddy                                           | {"level":"debug","ts":1727856242.914701,"logger":"tls.handshake","msg":"choosing certificate","identifier":"sentry.mydomain.com","num_choices":1}
caddy                                           | {"level":"debug","ts":1727856242.9147315,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"sentry.mydomain.com","subjects":["sentry.mydomain.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1727856242.9147413,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"****","remote_port":"34752","subjects":["sentry.mydomain.com"],"managed":true,"expiration":1733305272,"hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1727856242.954508,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"nginx:80","total_upstreams":1}
relay-1                                         | 2024-10-02T08:04:02.956362Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_request: started processing request
relay-1                                         | 2024-10-02T08:04:02.956872Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-10-02T08:04:02.956940Z TRACE request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: relay_server::endpoints::common: Sending envelope to project cache for V1 buffer
relay-1                                         | 2024-10-02T08:04:02.956976Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=0 ms status=200
relay-1                                         | 2024-10-02T08:04:02.957085Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-10-02T08:04:02.957095Z TRACE relay_server::services::project_cache: Sending envelope to processor
relay-1                                         | 2024-10-02T08:04:02.957102Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
nginx-1                                         | 86.126.135.131 - - [02/Oct/2024:08:04:02 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.12.0" "86.126.135.131"
relay-1                                         | 2024-10-02T08:04:02.957192Z TRACE relay_server::services::processor: Processing error group
relay-1                                         | 2024-10-02T08:04:02.957204Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-10-02T08:04:02.957221Z TRACE relay_server::services::processor::event: processing json event
relay-1                                         | 2024-10-02T08:04:02.957498Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-10-02T08:04:02.957512Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
caddy                                           | {"level":"debug","ts":1727856242.9574447,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"nginx:80","duration":0.00277329,"request":{"remote_ip":"****","remote_port":"34752","client_ip":"****","proto":"HTTP/1.1","method":"POST","host":"nginx:80","uri":"/api/3/envelope/","headers":{"Content-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["sentry.mydomain.com"],"X-Forwarded-For":["86.126.135.131"],"Accept-Encoding":["identity"],"Content-Length":["2077"],"Content-Type":["application/x-sentry-envelope"],"User-Agent":["sentry.python/2.12.0"],"X-Sentry-Auth":["Sentry sentry_key=6c77219228f5973a3f9efc00978f35ae, sentry_version=7, sentry_client=sentry.python/2.12.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"sentry.mydomain.com"}},"headers":{"Access-Control-Expose-Headers":["x-sentry-error,x-sentry-rate-limits,retry-after"],"Date":["Wed, 02 Oct 2024 08:04:02 GMT"],"Connection":["keep-alive"],"Access-Control-Allow-Origin":["*"],"Vary":["origin","access-control-request-method","access-control-request-headers"],"Cross-Origin-Resource-Policy":["cross-origin"],"Server":["nginx"],"Content-Type":["application/json"],"Content-Length":["41"]},"status":200}
relay-1                                         | 2024-10-02T08:04:02.957754Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-10-02T08:04:02.958191Z TRACE relay_server::services::processor: sending envelope to sentry endpoint
relay-1                                         | 2024-10-02T08:04:02.958747Z TRACE relay_server::services::cogs: recording measurement: CogsMeasurement { resource: Relay, feature: Errors, value: Time(1.421747ms) }
web-1                                           | 08:04:02 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1                                           | 08:04:02 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='***' user_agent='sentry.python/2.12.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.018032312393188477 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
relay-1                                         | 2024-10-02T08:04:07.441580Z TRACE relay_server::services::project_cache: handle_periodic_unspool

If this is not useful I might also be able to provide the envelope body payload from a project that uses sentry-nextjs SDK "@sentry/nextjs": "^8.28.0",. Would that be more useful? I chose the Python example for simplicity

stayallive commented 1 month ago

As long as we have an envelope that is causing the error on your side it is helpful no matter the SDK that generated it. Although it might be a Python specific issue but if you are able to generate a envelope that crashes on this error it would be very helpful for debugging.

stayallive commented 1 month ago

So scratch that. This is a debug error confirmed by someone on the Relay team. There is no effect on the error/transaction that is being ingested that should still come through just fine. So the problem you are facing is not related to this "error":

relay-1                                         | 2024-10-02T08:04:02.957512Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string

It will still be fixed in an upcoming release but I have been assured events should still come through even though this message is present 😄

razvan286 commented 1 month ago

I am triggering an intentional error from a NextJS client. I am using @sentry/nextjs version 8.28.0. Triggered error:

onClick={() => {
                const error = new Error('test sentry');
                Sentry.captureException(error);
                throw error;
              }}

cat next.config.mjs:

const SentryWebpackPluginOptions = {
  org: 'sentry',
  project: 'myproject',
  sentryUrl: 'https://sentry.mydomain.com/',
  silent: !process.env.CI,
  hideSourceMaps: true,
  sourcemaps: {
    // Do not upload sourcemaps if not in CI
    disable: !process.env.CI,
  },
  disableLogger: true,
};

cat sentry.client.config.js:

Sentry.init({
    dsn: process.env.NEXT_PUBLIC_SENTRY_DSN,
    tracesSampleRate: 1,
    debug: true,
    enabled: true,
    environment: process.env.NEXT_PUBLIC_SENTRY_ENVIRONMENT,

    // Session Replay - replayIntegration()
    replaysSessionSampleRate: 0.1,
    replaysOnErrorSampleRate: 1.0,
  });
}

I need to mention that there have been no changes to the self-hosted Sentry setup mentioned above. Worth mentioning inside the sentry/sentry.conf.py file the CSRF_TRUSTED_ORIGINS list is: CSRF_TRUSTED_ORIGINS = ["http://sentry.mydomain.com", "https://sentry.mydomain.com"]

I throw the error from the NextJS client that is run locally with SSL and I get the error:

web-1                                           | 09:37:42 [WARNING] django.security.csrf: Forbidden (Origin checking failed - https://myapp.com:3001/ does not match any trusted origins.): /api/2/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/2/envelope/'>)

Then, I add the origin https://myapp.com:3001 to the CSRF_TRUSTED_ORIGINS list. Do I need to add to this list all the client's origins? I will be using self-hosted sentry for multiple projects where each project has multiple environments for review, staging, and production.

Moreover, I run the ./install.sh I get rid of the error above and I get again the error mentioned in my last comment Forbidden (CSRF cookie not set.).

Current value of the CSRF_TRUSTED_ORIGINS list:

CSRF_TRUSTED_ORIGINS = ["http://sentry.mydomain.com", "https://sentry.mydomain.com", "https://myapp.com:3001"]

I am triggering the error again from the NextJS client and here are the docker compose logs:

caddy                                           | {"level":"debug","ts":1727948428.9880548,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"nginx:80","total_upstreams":1}
relay-1                                         | 2024-10-03T09:40:28.988981Z DEBUG request{method=POST uri=/api/2/envelope/?sentry_key=****&sentry_version=7&sentry_client=sentry.javascript.nextjs%2F8.28.0 version=HTTP/1.1}: tower_http::trace::on_request: started processing request
relay-1                                         | 2024-10-03T09:40:28.989462Z TRACE request{method=POST uri=/api/2/envelope/?sentry_key=****&sentry_version=7&sentry_client=sentry.javascript.nextjs%2F8.28.0 version=HTTP/1.1}: relay_server::endpoints::common: Sending envelope to project cache for V1 buffer
relay-1                                         | 2024-10-03T09:40:28.989524Z DEBUG request{method=POST uri=/api/2/envelope/?sentry_key=****&sentry_version=7&sentry_client=sentry.javascript.nextjs%2F8.28.0 version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=0 ms status=200
relay-1                                         | 2024-10-03T09:40:28.989644Z TRACE relay_server::services::project_cache: Sending envelope to processor
relay-1                                         | 2024-10-03T09:40:28.989738Z TRACE relay_server::services::processor: Processing replay group
nginx-1                                         | 86.126.135.131 - - [03/Oct/2024:09:40:28 +0000] "POST /api/2/envelope/?sentry_key=****&sentry_version=7&sentry_client=sentry.javascript.nextjs%2F8.28.0 HTTP/1.1" 200 41 "https://myapp.com:3001/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "****"
caddy                                           | {"level":"debug","ts":1727948428.9902182,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"nginx:80","duration":0.001942048,"request":{"remote_ip":"****","remote_port":"42116","client_ip":"****","proto":"HTTP/3.0","method":"POST","host":"nginx:80","uri":"/api/2/envelope/?sentry_key=****&sentry_version=7&sentry_client=sentry.javascript.nextjs%2F8.28.0","headers":{"Sec-Ch-Ua":["\"Not;A=Brand\";v=\"24\", \"Chromium\";v=\"128\""],"Referer":["https://myapp.com:3001/"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["cors"],"Origin":["https://myapp.com:3001"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["86.126.135.131"],"Content-Length":["3712"],"Sec-Fetch-Site":["cross-site"],"Accept-Language":["en-US,en;q=0.9"],"Priority":["u=1, i"],"Sec-Fetch-Dest":["empty"],"Sec-Ch-Ua-Platform":["\"Linux\""],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"],"Accept":["*/*"],"X-Forwarded-Host":["sentry.mydomain.com"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"sentry.mydomain.com"}},"headers":{"Content-Type":["application/json"],"Connection":["keep-alive"],"Access-Control-Allow-Origin":["*"],"Vary":["origin","access-control-request-method","access-control-request-headers"],"Cross-Origin-Resource-Policy":["cross-origin"],"Server":["nginx"],"Date":["Thu, 03 Oct 2024 09:40:28 GMT"],"Content-Length":["41"],"Access-Control-Expose-Headers":["x-sentry-error,x-sentry-rate-limits,retry-after"]},"status":200}
relay-1                                         | 2024-10-03T09:40:28.991764Z TRACE relay_server::services::processor: sending envelope to sentry endpoint
relay-1                                         | 2024-10-03T09:40:28.992506Z TRACE relay_server::services::cogs: recording measurement: CogsMeasurement { resource: Relay, feature: Replays, value: Time(2.662616ms) }
web-1                                           | 09:40:29 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/2/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/2/envelope/'>)
web-1                                           | 09:40:29 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/2/envelope/' caller_ip='****' user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.01250767707824707 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

Moreover, here are the envelope request details.

Query String Parameters:

sentry_key=********************&sentry_version=7&sentry_client=sentry.javascript.nextjs%2F8.28.0

Request Payload:

{"event_id":"8e619d9fe21246e181f8e4e8971820e8","sent_at":"2024-10-03T09:40:25.958Z","sdk":{"name":"sentry.javascript.nextjs","version":"8.28.0"},"trace":{"environment":"local","release":"Z-bJmL2JCKGoikSjFYdui","public_key":"****","trace_id":"88587b8311c64a3d818f4fb2b77a4c65","replay_id":"b46f824b0c8a4b97b2302c71ee8a9f77","sample_rate":"1","transaction":"/","sampled":"true"}}
{"type":"event"}
{"exception":{"values":[{"type":"Error","value":"test sentry","stacktrace":{"frames":[{"filename":"app:///_next/static/chunks/pages/_app-41ef0ce7ccadb0eb.js","function":"HTMLDivElement.r","in_app":true,"lineno":13,"colno":3310},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"nD","in_app":false,"lineno":9,"colno":20627},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"nU","in_app":false,"lineno":9,"colno":20843},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"ro","in_app":false,"lineno":9,"colno":35174},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"eF","in_app":false,"lineno":9,"colno":14048},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"oP","in_app":false,"lineno":9,"colno":96292},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"?","in_app":false,"lineno":9,"colno":39339},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"rn","in_app":false,"lineno":9,"colno":33883},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"re","in_app":false,"lineno":9,"colno":33469},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"?","in_app":false,"lineno":9,"colno":33370},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"eH","in_app":false,"lineno":9,"colno":15064},{"filename":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","function":"Object.eU","in_app":false,"lineno":9,"colno":14910},{"filename":"app:///_next/static/chunks/pages/index-75192777e856c338.js","function":"onClick","in_app":true,"lineno":1,"colno":48607}]},"mechanism":{"type":"generic","handled":true}}]},"level":"error","event_id":"8e619d9fe21246e181f8e4e8971820e8","platform":"javascript","request":{"url":"https://myapp.com:3001/","headers":{"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"}},"timestamp":1727948425.945,"environment":"local","release":"Z-bJmL2JCKGoikSjFYdui","sdk":{"integrations":["InboundFilters","FunctionToString","BrowserApiErrors","Breadcrumbs","GlobalHandlers","LinkedErrors","Dedupe","HttpContext","BrowserTracing","NextjsClientStackFrameNormalization","Replay"],"name":"sentry.javascript.nextjs","version":"8.28.0","packages":[{"name":"npm:@sentry/nextjs","version":"8.28.0"},{"name":"npm:@sentry/react","version":"8.28.0"}]},"contexts":{"trace":{"trace_id":"88587b8311c64a3d818f4fb2b77a4c65","span_id":"871e6713ffc9e162"},"react":{"version":"18.2.0"}},"transaction":"/","breadcrumbs":[{"timestamp":1727948411.554,"category":"navigation","data":{"from":"/","to":"/"}},{"timestamp":1727948411.629,"category":"navigation","data":{"from":"/","to":"/"}},{"timestamp":1727948411.713,"category":"fetch","data":{"method":"GET","url":"/api/origin","__span":"9e4b691c8890f10c","status_code":200,"response_body_size":2},"type":"http"},{"timestamp":1727948411.825,"category":"fetch","data":{"method":"GET","url":"/api/getNewsFiles?locale=en","__span":"aa43f78a880a0747","status_code":200,"response_body_size":119},"type":"http"},{"timestamp":1727948411.891,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/support.json","__span":"a97ae11457595e7e","status_code":200},"type":"http"},{"timestamp":1727948411.908,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en.json","__span":"8446b4d8908eb378","status_code":200},"type":"http"},{"timestamp":1727948411.962,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/news.json","__span":"af003a74b07f009c","status_code":200},"type":"http"},{"timestamp":1727948411.977,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/webshops.json","__span":"818fd49705d88a07","status_code":200},"type":"http"},{"timestamp":1727948412.007,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/developer.json","__span":"83b3ede08e2ed3f5","status_code":200},"type":"http"},{"timestamp":1727948412.013,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/downloads/nl.json?country=nl","__span":"b785f28d54d5c594","status_code":200},"type":"http"},{"timestamp":1727948412.036,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/demo/address-validation/nl.json?country=nl","__span":"a39ccd8bc42c623c","status_code":200},"type":"http"},{"timestamp":1727948412.069,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/login.json","__span":"9f57ee708e7e95ca","status_code":200},"type":"http"},{"timestamp":1727948412.071,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/pricing.json","__span":"84bbef7efcd14edf","status_code":200},"type":"http"},{"timestamp":1727948412.073,"category":"xhr","data":{"method":"POST","url":"https://plausible.mydomain.com/api/event","status_code":202,"request_body_size":74,"response_body_size":2},"type":"http"},{"timestamp":1727948412.901,"category":"sentry.transaction","event_id":"fd08c32f1b8f4af38f8e6ea56480bef6","message":"fd08c32f1b8f4af38f8e6ea56480bef6"},{"timestamp":1727948424.478,"category":"fetch","data":{"method":"GET","url":"/_next/data/Z-bJmL2JCKGoikSjFYdui/en/login.json","__span":"9dad2841cd94f874","status_code":200},"type":"http"},{"timestamp":1727948425.933,"category":"ui.click","message":"article.flex.flex-col.gap-3.self-start > h1"}],"tags":{"replayId":"b46f824b0c8a4b97b2302c71ee8a9f77"},"debug_meta":{"images":[{"type":"sourcemap","code_file":"app:///_next/static/chunks/pages/_app-41ef0ce7ccadb0eb.js","debug_id":"063c4d59-8a0b-4bb0-89bb-6e7c6cbe7479"},{"type":"sourcemap","code_file":"app:///_next/static/chunks/framework-9bcc3cdf64e58f17.js","debug_id":"381f689e-7c3e-4932-ac08-cb4784daffcd"},{"type":"sourcemap","code_file":"app:///_next/static/chunks/pages/index-75192777e856c338.js","debug_id":"189b7e07-ca79-4016-9327-5824096a6b34"}]}}

Response: {"id":"8e619d9fe21246e181f8e4e8971820e8"}

Request URL: `https://sentry.mydomain.com/api/2/envelope/?sentry_key=****&sentry_version=7&sentry_client=sentry.javascript.nextjs%2F8.28.0``

Request Method: POST

Status Code: 200 OK

I am not sure how to debug this in more depth, if you have any suggestions I will provide you with even more information.igg

aldy505 commented 1 month ago

Then, I add the origin https://myapp.com:3001 to the CSRF_TRUSTED_ORIGINS list. Do I need to add to this list all the client's origins? I will be using self-hosted sentry for multiple projects where each project has multiple environments for review, staging, and production.

No, the CSRF_TRUSTED_ORIGINS should be only set to the domain you're accessing your Sentry web UI from.

Do you have any additional reverse proxy in front of your Sentry instance? I don't face this issue because I only expose the ingest endpoint publicly (see docs), everything other than that requires a private network. You have Caddy. Just forget everything I said.

kvdb commented 1 month ago

Is there anything we can do (test, investigate) to help someone implement the bugfix?

aldy505 commented 2 weeks ago

Hi @kvdb sorry for the very late reply. Are you still experiencing this problem?

razvan286 commented 1 week ago

The issue has been fixed after upgrading to version 24.11.0. We appreciate your help!