User Settings > Notifications produces "Oops! Something went wrong" error

[csprocket777]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

What are you trying to accomplish?

I am trying to adjust the notification settings for my user account.

Replication steps:

1. Go to my user settings (Upper-left menu, "User settings")
2. Select "Notifications"
3. Click the gear icon to the right of "Issue Alerts" (really it could be any of them)

I then see the following verbiage in the page:

Oops! Something went wrong
It looks like you've hit an issue in our client application. Don't worry though! We use Sentry to monitor Sentry and it's likely we're already looking into this!

If you're daring, you may want to try the following:

Give it a few seconds and reload the page.
If all else fails, [contact us](https://github.com/getsentry/sentry/issues/new/choose) with more details.

I suspect it might be something to do with my nginx configuration but I don't see any errors in my nginx error log and I can't seem to see any issues being reported when I look at the Sentry logs using docker compose logs

Environment context:

OS Info: CentOS Linux release 7.9.2009 (Core)

Docker Info:

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:11:35 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:10:36 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I have an nginx reverse proxy setup in order to support HTTPS connections.

Nginx.conf: (This is working afaik)

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    # set REMOTE_ADDR from any internal proxies
    # see http://nginx.org/en/docs/http/ngx_http_realip_module.html
    set_real_ip_from 127.0.0.1;
    set_real_ip_from 10.0.0.0/8;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;

    include mime.types;
    default_type application/octet-stream;
    sendfile on;
    keepalive_timeout 65;

    server {
        listen      80;
        server_name [my-domain.com];

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        location / {
            if ($request_method = GET) {
                rewrite  ^ https://$host$request_uri? permanent;
            }
        proxy_pass http://localhost:9000;
            return 405;
        }
    }

    server {
        listen 443 ssl;
        http2  on;

        server_name [my-domain.com];
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ssl_certificate     /etc/ssl/certs/[bundledDomainCert].bundle.crt;
        ssl_certificate_key /etc/ssl/certs/[domainKey].key;

        ssl_session_timeout 1d;
        ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
        ssl_session_tickets off;

        # modern configuration
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;

        # HSTS (ngx_http_headers_module is required) (63072000 seconds)
        add_header Strict-Transport-Security "max-age=63072000" always;

        # OCSP stapling
        ssl_stapling        off;
        ssl_stapling_verify off;

        # replace with the IP address of your resolver
        resolver 127.0.0.1;

        location / {
            proxy_set_header    Host            $host;
            proxy_set_header    X-Real-IP       $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
            proxy_set_header    X-Forwarded-Proto   $scheme;
            proxy_set_header    X-Forwarded-Host    $server_name;
            proxy_set_header    X-Forwarded-Ssl     on;
            proxy_set_header    Upgrade         $http_upgrade;
            proxy_set_header    Connection      "upgrade";
            proxy_http_version  1.1;
            proxy_redirect      off;

            # keepalive + raven.js is a disaster
            keepalive_timeout 0;

            # use very aggressive timeouts
            proxy_read_timeout 5s;
            proxy_send_timeout 5s;
            send_timeout 5s;
            resolver_timeout 5s;
            client_body_timeout 5s;

            # buffer larger messages
            client_max_body_size 5m;
            client_body_buffer_size 100k;

            proxy_pass http://localhost:9000;
        }
    }
}

config.yml file:

# While a lot of configuration in Sentry can be changed via the UI, for all
# new-style config (as of 8.0) you can also declare values here in this file
# to enforce defaults or to ensure they cannot be changed via the UI. For more
# information see the Sentry documentation.

###############
# Mail Server #
###############

# mail.backend: 'smtp'  # Use dummy if you want to disable email entirely
mail.host: 'smtp'
# mail.port: 25
# mail.username: ''
# mail.password: ''
# mail.use-tls: false
# mail.use-ssl: false

# NOTE: The following 2 configs (mail.from and mail.list-namespace) are set
#       through SENTRY_MAIL_HOST in sentry.conf.py so remove those first if
#       you want your values in this file to be effective!

# The email address to send on behalf of
# mail.from: 'root@localhost' or ...
# mail.from: 'System Administrator <root@localhost>'

# The mailing list namespace for emails sent by this Sentry server.
# This should be a domain you own (often the same domain as the domain
# part of the `mail.from` configuration parameter value) or `localhost`.
# mail.list-namespace: 'localhost'

# If you'd like to configure email replies, enable this.
# mail.enable-replies: true

# When email-replies are enabled, this value is used in the Reply-To header
# mail.reply-hostname: ''

# If you're using mailgun for inbound mail, set your API key and configure a
# route to forward to /api/hooks/mailgun/inbound/
# Also don't forget to set `mail.enable-replies: true` above.
# mail.mailgun-api-key: ''

###################
# System Settings #
###################

# If this file ever becomes compromised, it's important to generate a new key.
# Changing this value will result in all current sessions being invalidated.
# A new key can be generated with `$ sentry config generate-secret-key`
system.secret-key: '[redacted]'

# The ``redis.clusters`` setting is used, unsurprisingly, to configure Redis
# clusters. These clusters can be then referred to by name when configuring
# backends such as the cache, digests, or TSDB backend.
# redis.clusters:
#   default:
#     hosts:
#       0:
#         host: 127.0.0.1
#         port: 6379

################
# File storage #
################

# Uploaded media uses these `filestore` settings. The available
# backends are either `filesystem` or `s3`.

filestore.backend: 'filesystem'
filestore.options:
  location: '/data/files'
dsym.cache-path: '/data/dsym-cache'
releasefile.cache-path: '/data/releasefile-cache'

# filestore.backend: 's3'
# filestore.options:
#   access_key: 'AKIXXXXXX'
#   secret_key: 'XXXXXXX'
#   bucket_name: 's3-bucket-name'

system.url-prefix: 'https://[my-domain].com'
system.internal-url-prefix: 'http://web:9000'
symbolicator.enabled: true
symbolicator.options:
  url: "http://symbolicator:3021"

transaction-events.force-disable-internal-project: true

######################
# GitHub Integration #
######################

# github-login.extended-permissions: ['repo']
# github-app.id: GITHUB_APP_ID
# github-app.name: 'GITHUB_APP_NAME'
# github-app.webhook-secret: 'GITHUB_WEBHOOK_SECRET' # Use only if configured in GitHub
# github-app.client-id: 'GITHUB_CLIENT_ID'
# github-app.client-secret: 'GITHUB_CLIENT_SECRET'
# github-app.private-key: |
#   -----BEGIN RSA PRIVATE KEY-----
#   privatekeyprivatekeyprivatekeyprivatekey
#   privatekeyprivatekeyprivatekeyprivatekey
#   privatekeyprivatekeyprivatekeyprivatekey
#   privatekeyprivatekeyprivatekeyprivatekey
#   privatekeyprivatekeyprivatekeyprivatekey
#   -----END RSA PRIVATE KEY-----

#####################
# Slack Integration #
#####################

# Refer to https://develop.sentry.dev/integrations/slack/ for setup instructions.

# slack.client-id: <'client id'>
# slack.client-secret: <client secret>
# slack.signing-secret: <signing secret>
## If legacy-app is True use verfication-token instead of signing-secret
# slack.verification-token: <verification token>

sentry.conf.py:

# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.

from sentry.conf.server import *  # NOQA

# Generously adapted from pynetlinux: https://github.com/rlisagor/pynetlinux/blob/e3f16978855c6649685f0c43d4c3fcf768427ae5/pynetlinux/ifconfig.py#L197-L223
def get_internal_network():
    import ctypes
    import fcntl
    import math
    import socket
    import struct

    iface = b"eth0"
    sockfd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    ifreq = struct.pack(b"16sH14s", iface, socket.AF_INET, b"\x00" * 14)

    try:
        ip = struct.unpack(
            b"!I", struct.unpack(b"16sH2x4s8x", fcntl.ioctl(sockfd, 0x8915, ifreq))[2]
        )[0]
        netmask = socket.ntohl(
            struct.unpack(b"16sH2xI8x", fcntl.ioctl(sockfd, 0x891B, ifreq))[2]
        )
    except IOError:
        return ()
    base = socket.inet_ntoa(struct.pack(b"!I", ip & netmask))
    netmask_bits = 32 - int(round(math.log(ctypes.c_uint32(~netmask).value + 1, 2), 1))
    return "{0:s}/{1:d}".format(base, netmask_bits)

INTERNAL_SYSTEM_IPS = (get_internal_network(),)

DATABASES = {
    "default": {
        "ENGINE": "sentry.db.postgres",
        "NAME": "postgres",
        "USER": "postgres",
        "PASSWORD": "",
        "HOST": "postgres",
        "PORT": "",
    }
}

# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True

# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings

###########
# General #
###########

# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True

SENTRY_OPTIONS["system.event-retention-days"] = int(
    env("SENTRY_EVENT_RETENTION_DAYS", "90")
)

#########
# Redis #
#########

# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB

SENTRY_OPTIONS["redis.clusters"] = {
    "default": {
        "hosts": {0: {"host": "redis", "password": "", "port": "6379", "db": "0"}}
    }
}

#########
# Queue #
#########

# See https://develop.sentry.dev/services/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.

rabbitmq_host = None
if rabbitmq_host:
    BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
        username="guest", password="guest", host=rabbitmq_host, vhost="/"
    )
else:
    BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
        **SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
    )

#########
# Cache #
#########

# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
        "LOCATION": ["memcached:11211"],
        "TIMEOUT": 3600,
    }
}

# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"

DEFAULT_KAFKA_OPTIONS = {
    "bootstrap.servers": "kafka:9092",
    "message.max.bytes": 50000000,
    "socket.timeout.ms": 1000,
}

SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream"
SENTRY_EVENTSTREAM_OPTIONS = {"producer_configuration": DEFAULT_KAFKA_OPTIONS}

KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS

###############
# Rate Limits #
###############

# Rate limits apply to notification handlers and are enforced per-project
# automatically.

SENTRY_RATELIMITER = "sentry.ratelimits.redis.RedisRateLimiter"

##################
# Update Buffers #
##################

# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)

SENTRY_BUFFER = "sentry.buffer.redis.RedisBuffer"

##########
# Quotas #
##########

# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.

SENTRY_QUOTAS = "sentry.quotas.redis.RedisQuota"

########
# TSDB #
########

# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.

SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"

#########
# SNUBA #
#########

SENTRY_SEARCH = "sentry.search.snuba.EventsDatasetSnubaSearchBackend"
SENTRY_SEARCH_OPTIONS = {}
SENTRY_TAGSTORE_OPTIONS = {}

###########
# Digests #
###########

# The digest backend powers notification summaries.

SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed in order to prevent https://github.com/getsentry/sentry/blob/c6f9660e37fcd9c1bbda8ff4af1dcfd0442f5155/src/sentry/services/http.py#L70
    "uwsgi-socket": None,
    "so-keepalive": True,
    # Keep this between 15s-75s as that's what Relay supports
    "http-keepalive": 15,
    "http-chunked-input": True,
    # the number of web workers
    "workers": 3,
    "threads": 4,
    "memory-report": False,
    # Some stuff so uwsgi will cycle workers sensibly
    "max-requests": 100000,
    "max-requests-delta": 500,
    "max-worker-lifetime": 86400,
    # Duplicate options from sentry default just so we don't get
    # bit by sentry changing a default value that we depend on.
    "thunder-lock": True,
    "log-x-forwarded-for": False,
    "buffer-size": 32768,
    "limit-post": 209715200,
    "disable-logging": True,
    "reload-on-rss": 600,
    "ignore-sigpipe": True,
    "ignore-write-errors": True,
    "disable-write-exception": True,
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

# End of SSL/TLS settings

########
# Mail #
########

SENTRY_OPTIONS["mail.list-namespace"] = env('SENTRY_MAIL_HOST', 'localhost')
SENTRY_OPTIONS["mail.from"] = f"sentry@{SENTRY_OPTIONS['mail.list-namespace']}"

############
# Features #
############

SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            "organizations:discover",
            "organizations:events",
            "organizations:global-views",
            "organizations:incidents",
            "organizations:integrations-issue-basic",
            "organizations:integrations-issue-sync",
            "organizations:invite-members",
            "organizations:metric-alert-builder-aggregate",
            "organizations:sso-basic",
            "organizations:sso-rippling",
            "organizations:sso-saml2",
            "organizations:performance-view",
            "organizations:advanced-search",
            "projects:custom-inbound-filters",
            "projects:data-forwarding",
            "projects:discard-groups",
            "projects:plugins",
            "projects:rate-limits",
            "projects:servicehooks",
        )
    }
)

#######################
# MaxMind Integration #
#######################

GEOIP_PATH_MMDB = '/geoip/GeoLite2-City.mmdb'

#########################
# Bitbucket Integration #
#########################

# BITBUCKET_CONSUMER_KEY = 'YOUR_BITBUCKET_CONSUMER_KEY'
# BITBUCKET_CONSUMER_SECRET = 'YOUR_BITBUCKET_CONSUMER_SECRET'

How are you getting stuck?

I don't know how to troubleshoot this or resolve it and could use some help or pointers.

Where in the product are you?

Settings

Link

No response

DSN

No response

Version

23.11.2

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[hubertdeng123]

Are your other pages loading fine? I don't necessarily think this is an nginx config issue. It's concerning that you're not seeing any logs, I'd expect at least a 500 error or something in your web container logs.

[csprocket777]

I have not run across any other issues with any other page.

[csprocket777]

What log commands should i use to examine this?

[azaslavsky]

Can you open devtools in your browser of choice, and note 2 things:

1. What network calls occur when you click the settings icon that causes this error (Network tab in Chrome devtools). In particular, we're looking for calls that result in 4xx or 5xx responses.
2. More generally, do you see any errors reported in the console when you do this?

The issue seems to be client-side - either it's not hitting the server, or else is somehow mishandling this communication. If we can get some more information from the client, that would go a long way in helping get to the bottom of this.

[zKoz210]

1. Go to https://DOMAIN/settings/account/notifications/
2. Click Issue Workflow
3. Open https://DOMAIN/settings/account/notifications/workflow/
4. API Result https://DOMAIN/api/0/users/me/notifications/workflow/ is 404

[Emkas]

We're facing the same issue after upgrade to 23.11.2.

Based on what I read in #2617 this problem can be related. And maybe it will be fixed in 23.12.0.

[hubertdeng123]

After upgrading to 23.12.0, is this still an issue?

[Emkas]

After upgrading to 23.12.0, is this still an issue?

No, everything works fine in 23.12.0.