search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.86k stars 1.77k forks source link

SnubaError: HTTPConnectionPool(host='snuba-api', port=1218): Read timed out. (read timeout=30) #2151

Open AlexanderRydberg opened 1 year ago

AlexanderRydberg commented 1 year ago

Self-Hosted Version

23.4.0

CPU Architecture

x86_64

Docker Version

20.10.7

Docker Compose Version

2.3.3

Steps to Reproduce

This one is a bit hard to describe since I cant really reproduce it. Or atleast I cant see the pattern to reproduce it. The first time we saw this error was on 22.3.0. And it has happened quite random since. Sometimes when navigating to an exception and/or navigating to /issues/ it times out. And Sentry throws exceptions.

After reload it works.

And this is a really bad written ticket, But ask away and Ill try to answer with some help.

Expected Result

Can I write "For it not to time out" :) ?

Actual Result

When it times out we see this: Issues and exception ls -1 sentry_install_log-*.txt | tail -1 | xargs cat `▶ Parsing command line ...

▶ Detecting Docker platform Detected Docker platform is linux/amd64

▶ Initializing Docker Compose ...

▶ Setting up error handling ... Sending build context to Docker daemon 2.048kB Step 1/4 : FROM debian:bullseye-slim bullseye-slim: Pulling from library/debian 9e3ea8720c6d: Pulling fs layer 9e3ea8720c6d: Verifying Checksum 9e3ea8720c6d: Download complete 9e3ea8720c6d: Pull complete Digest: sha256:f4da3f9b18fc242b739807a0fb3e77747f644f2fb3f67f4403fafce2286b431a Status: Downloaded newer image for debian:bullseye-slim ---> 07df01f107e8 Step 2/4 : LABEL MAINTAINER="oss@sentry.io" ---> Running in b88446120cac Removing intermediate container b88446120cac ---> a2b465280135 Step 3/4 : RUN set -x && apt-get update && DEBIAN_FRONTEND=noninteractive ap t-get install -y --no-install-recommends jq && apt-get clean && rm -rf /var/ lib/apt/lists/* ---> Running in b6cea2e78034

Hey, so ... we would love to automatically find out about issues with your Sentry instance so that we can improve the product. Turns out there is an app for that, called Sentry. Would you be willing to let us automatically send data about your instance upstream to Sentry for development and debugging purposes?

y / yes / 1 n / no / 0

(Btw, we send this to our own self-hosted Sentry instance, not to Sentry SaaS, so that we can be in this together.)

Here's the info we may collect:

Thirty (30) day retention. No marketing. Privacy policy at sentry.io/privacy.

y or n? Thank you. To avoid this prompt in the future, use one of these flags:

--report-self-hosted-issues --no-report-self-hosted-issues

or set the REPORT_SELF_HOSTED_ISSUES environment variable:

REPORT_SELF_HOSTED_ISSUES=1 to send data REPORT_SELF_HOSTED_ISSUES=0 to not send data

latest: Pulling from getsentry/sentry-cli f7dab3ab2d6e: Pulling fs layer 8bd10b77a777: Pulling fs layer 80aeafe5b1af: Pulling fs layer e75529525eaf: Pulling fs layer 39f5ba1d1194: Pulling fs layer e75529525eaf: Waiting 39f5ba1d1194: Waiting 8bd10b77a777: Verifying Checksum 8bd10b77a777: Download complete f7dab3ab2d6e: Verifying Checksum f7dab3ab2d6e: Download complete 80aeafe5b1af: Verifying Checksum 80aeafe5b1af: Download complete e75529525eaf: Download complete 39f5ba1d1194: Verifying Checksum 39f5ba1d1194: Download complete f7dab3ab2d6e: Pull complete 8bd10b77a777: Pull complete 80aeafe5b1af: Pull complete e75529525eaf: Pull complete 39f5ba1d1194: Pull complete Digest: sha256:485545f1581f40eae68d5eca153eb80868e47542144351381a7b98ecd5d82a91 Status: Downloaded newer image for getsentry/sentry-cli:latest docker.io/getsentry/sentry-cli:latest

▶ Checking for latest commit ... skipped

▶ Checking minimum requirements ... Found Docker version 20.10.7 Found Docker Compose version 2.3.3 Unable to find image 'busybox:latest' locally latest: Pulling from library/busybox 4b35f584bb4f: Pulling fs layer 4b35f584bb4f: Verifying Checksum 4b35f584bb4f: Download complete 4b35f584bb4f: Pull complete Digest: sha256:b5d6fe0712636ceb7430189de28819e195e8966372edfc2d9409d79402a0dc16 Status: Downloaded newer image for busybox:latest

▶ Turning things off ... sentry-self-hosted Warning: No resource found to remove

▶ Creating volumes for persistent storage ... Created sentry-clickhouse. Created sentry-data. Created sentry-kafka. Created sentry-postgres. Created sentry-redis. Created sentry-symbolicator. Created sentry-zookeeper.

▶ Ensuring files from examples ... sentry/sentry.conf.py already exists, skipped creation. sentry/config.yml already exists, skipped creation. symbolicator/config.yml already exists, skipped creation.

▶ Ensuring Relay credentials ... relay/config.yml already exists, skipped creation. relay/credentials.json already exists, skipped creation.

▶ Generating secret key ...

▶ Fetching and updating Docker images ... 23.4.0: Pulling from getsentry/sentry Digest: sha256:bd4d3e2b3c868eefcb762cbc879474321e0f88dece78c56130dfaee6a2d5396a Status: Downloaded newer image for getsentry/sentry:23.4.0 docker.io/getsentry/sentry:23.4.0

▶ Building and tagging Docker images ...

Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 434B Step 1/2 : ARG BASE_IMAGE Step 2/2 : FROM ${BASE_IMAGE} ---> abe55fc6544d Successfully built abe55fc6544d Successfully tagged clickhouse-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 761B Step 1/8 : ARG BASE_IMAGE Step 2/8 : FROM ${BASE_IMAGE} ---> 569d5809d1ec Step 3/8 : USER 0 ---> Using cache ---> 582f9addb51d Step 4/8 : RUN if [ -z "${http_proxy}" ]; then echo "Acquire::http::proxy \"${ht tp_proxy}\";" >> /etc/apt/apt.conf; fi ---> Using cache ---> 699ffdbd00ad Step 5/8 : RUN if [ -z "${https_proxy}" ]; then echo "Acquire::https::proxy \"${ https_proxy}\";" >> /etc/apt/apt.conf; fi ---> Using cache ---> 31a67c8d03b2 Step 6/8 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/* ---> Using cache ---> 79f975e719ed Step 7/8 : COPY entrypoint.sh /entrypoint.sh ---> Using cache ---> 67b8cee247d7 Step 8/8 : ENTRYPOINT ["/entrypoint.sh"] ---> Using cache ---> 80bff9a8d618 Successfully built 80bff9a8d618 Successfully tagged sentry-cleanup-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 761B Step 1/8 : ARG BASE_IMAGE Step 2/8 : FROM ${BASE_IMAGE} ---> 0a2edfe3f08a Step 3/8 : USER 0 ---> Using cache ---> 9da537cabdbb Step 4/8 : RUN if [ -z "${http_proxy}" ]; then echo "Acquire::http::proxy \"${ht tp_proxy}\";" >> /etc/apt/apt.conf; fi ---> Using cache ---> 89d7bfbf81c4 Step 5/8 : RUN if [ -z "${https_proxy}" ]; then echo "Acquire::https::proxy \"${ https_proxy}\";" >> /etc/apt/apt.conf; fi ---> Using cache ---> d6c9cec5f130 Step 6/8 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/* ---> Using cache ---> 2d4225ef356d Step 7/8 : COPY entrypoint.sh /entrypoint.sh ---> Using cache ---> a1c24c67a621 Step 8/8 : ENTRYPOINT ["/entrypoint.sh"] ---> Using cache ---> 1ddf47982afb Successfully built 1ddf47982afb Successfully tagged symbolicator-cleanup-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them Sending build context to Docker daemon 6.319kB Step 1/5 : ARG SENTRY_IMAGE Step 2/5 : FROM ${SENTRY_IMAGE} ---> 66739bcbc315 Step 3/5 : COPY . /usr/src/sentry ---> Using cache ---> 1023a1961fa7 Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sen try/enhance-image.sh; fi ---> Using cache ---> 7500e8b43047 Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry /requirements.txt is deprecated, use sentry/enhance-image.sh - see https://githu b.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/s entry/requirements.txt; fi ---> Using cache ---> 569d5809d1ec Successfully built 569d5809d1ec Successfully tagged sentry-self-hosted-local:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l earn how to fix them

Docker images built.

▶ Downloading and installing wal2json ...

▶ Bootstrapping and migrating Snuba ... Network sentry-self-hosted_default Creating Network sentry-self-hosted_default Created Container sentry-self-hosted-redis-1 Creating Container sentry-self-hosted-zookeeper-1 Creating Container sentry-self-hosted-clickhouse-1 Creating Container sentry-self-hosted-clickhouse-1 Created Container sentry-self-hosted-redis-1 Created Container sentry-self-hosted-zookeeper-1 Created Container sentry-self-hosted-kafka-1 Creating Container sentry-self-hosted-kafka-1 Created Container sentry-self-hosted-clickhouse-1 Starting Container sentry-self-hosted-redis-1 Starting Container sentry-self-hosted-zookeeper-1 Starting Container sentry-self-hosted-clickhouse-1 Started Container sentry-self-hosted-redis-1 Started Container sentry-self-hosted-zookeeper-1 Started Container sentry-self-hosted-zookeeper-1 Waiting Container sentry-self-hosted-zookeeper-1 Healthy Container sentry-self-hosted-kafka-1 Starting Container sentry-self-hosted-kafka-1 Started 2023-05-03 13:07:00,267 Initializing Snuba... 2023-05-03 13:07:09,971 Snuba initialization took 9.705413776999194s 2023-05-03 13:07:10,552 Initializing Snuba... 2023-05-03 13:07:13,354 Snuba initialization took 2.8034897460001957s 2023-05-03 13:07:13,355 Attempting to connect to Kafka (attempt 0)... 2023-05-03 13:07:14,612 Connected to Kafka on attempt 0 2023-05-03 13:07:14,612 Creating Kafka topics... Container sentry-self-hosted-redis-1 Running Container sentry-self-hosted-clickhouse-1 Running Container sentry-self-hosted-zookeeper-1 Running Container sentry-self-hosted-kafka-1 Running Container sentry-self-hosted-zookeeper-1 Waiting Container sentry-self-hosted-zookeeper-1 Healthy 2023-05-03 13:07:18,001 Initializing Snuba... 2023-05-03 13:07:21,658 Snuba initialization took 3.658176357001139s 2023-05-03 13:07:22,167 Initializing Snuba... 2023-05-03 13:07:25,331 Snuba initialization took 3.1646578170002613s Finished running migrations

▶ Creating additional Kafka topics ...

▶ Ensuring proper PostgreSQL version ...

▶ Setting up / migrating database ... Container sentry-self-hosted-memcached-1 Creating Container sentry-self-hosted-smtp-1 Creating Container sentry-self-hosted-symbolicator-1 Creating Container sentry-self-hosted-postgres-1 Creating Container sentry-self-hosted-zookeeper-1 Running Container sentry-self-hosted-redis-1 Running Container sentry-self-hosted-kafka-1 Running Container sentry-self-hosted-clickhouse-1 Running Container sentry-self-hosted-snuba-consumer-1 Creating Container sentry-self-hosted-snuba-outcomes-consumer-1 Creating Container sentry-self-hosted-snuba-subscription-consumer-events-1 Creating Container sentry-self-hosted-snuba-sessions-consumer-1 Creating Container sentry-self-hosted-snuba-replacer-1 Creating Container sentry-self-hosted-snuba-transactions-consumer-1 Creating Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Creatin g Container sentry-self-hosted-snuba-api-1 Creating Container sentry-self-hosted-snuba-api-1 Created Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Created Container sentry-self-hosted-snuba-subscription-consumer-events-1 Created Container sentry-self-hosted-snuba-sessions-consumer-1 Created Container sentry-self-hosted-snuba-transactions-consumer-1 Created Container sentry-self-hosted-smtp-1 Created Container sentry-self-hosted-snuba-replacer-1 Created Container sentry-self-hosted-snuba-consumer-1 Created Container sentry-self-hosted-snuba-outcomes-consumer-1 Created Container sentry-self-hosted-memcached-1 Created Container sentry-self-hosted-symbolicator-1 Created Container sentry-self-hosted-postgres-1 Created Container sentry-self-hosted-symbolicator-1 Starting Container sentry-self-hosted-postgres-1 Starting Container sentry-self-hosted-memcached-1 Starting Container sentry-self-hosted-smtp-1 Starting Container sentry-self-hosted-zookeeper-1 Waiting Container sentry-self-hosted-zookeeper-1 Healthy Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-redis-1 Waiting Container sentry-self-hosted-clickhouse-1 Waiting Container sentry-self-hosted-kafka-1 Waiting Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Startin g Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-snuba-sessions-consumer-1 Starting Container sentry-self-hosted-snuba-outcomes-consumer-1 Starting Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-snuba-subscription-consumer-events-1 Starting Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-snuba-transactions-consumer-1 Starting Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-snuba-consumer-1 Starting Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-snuba-replacer-1 Starting Container sentry-self-hosted-redis-1 Healthy Container sentry-self-hosted-kafka-1 Healthy Container sentry-self-hosted-clickhouse-1 Healthy Container sentry-self-hosted-snuba-api-1 Starting Container sentry-self-hosted-postgres-1 Started Container sentry-self-hosted-snuba-transactions-consumer-1 Started Container sentry-self-hosted-snuba-subscription-consumer-events-1 Started Container sentry-self-hosted-symbolicator-1 Started Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Started Container sentry-self-hosted-memcached-1 Started Container sentry-self-hosted-snuba-replacer-1 Started Container sentry-self-hosted-snuba-outcomes-consumer-1 Started Container sentry-self-hosted-smtp-1 Started Container sentry-self-hosted-snuba-api-1 Started Container sentry-self-hosted-snuba-consumer-1 Started Container sentry-self-hosted-snuba-sessions-consumer-1 Started Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https:/ /github.com/getsentry/self-hosted#enhance-sentry-image 13:08:38 [INFO] sentry.plugins.github: apps-not-configured Running migrations for default Operations to perform: Apply all migrations: admin, auth, contenttypes, nodestore, replays, sentry, s essions, sites, social_auth Running migrations: No migrations to apply. Creating missing DSNs Correcting Group.num_comments counter

▶ Setting up GeoIP integration ... Setting up IP address geolocation ... IP address geolocation database already exists. IP address geolocation is not configured for updates. See https://develop.sentry.dev/self-hosted/geolocation/ for instructions. Error setting up IP address geolocation. `

Dont know if below log is relevant but sending it anyways sentry-self-hosted-snuba-api-1 logs return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,264 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,349 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,404 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,433 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,462 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,490 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,524 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,556 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,589 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,694 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,787 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,876 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' 2023-05-15 12:00:02,932 Allocation policy failed to update quota balance, this is a bug, fix it Traceback (most recent call last): File "/usr/src/snuba/./snuba/query/allocation_policies/__init__.py", line 200, in update_quota_balance return self._update_quota_balance(tenant_ids, result_or_error) File "/usr/src/snuba/./snuba/query/allocation_policies/errors_allocation_policy.py", line 191, in _update_quota_balance f"{self.rate_limit_prefix}-organization_id-{tenant_ids['organization_id']}", KeyError: 'organization_id' ...The work of process 20 is done. Seeya! worker 1 killed successfully (pid: 20) Respawned uWSGI worker 1 (new pid: 24) 2023-05-16 05:07:58,446 Initializing Snuba... WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x

Some cut outs from sentry-self-hosted-postgres-1: 2023-05-17 05:46:33.587 UTC [13848] ERROR: duplicate key value violates unique constraint "sentry_eventuser_project_id_ident_a9b6441c_uniq", 2023-05-17 05:46:33.587 UTC [13848] DETAIL: Key (project_id, ident)=(11, hayhgwovwol54wd4z3x2sl3b) already exists., 2023-05-17 06:01:54.484 UTC [14951] ERROR: duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq", 2023-05-17 06:01:54.484 UTC [14951] DETAIL: Key (project_id, environment_id)=(5, 4) already exists., 2023-05-17 06:01:54.570 UTC [14951] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq", 2023-05-17 06:01:54.570 UTC [14951] DETAIL: Key (group_id, release_id, environment)=(5121, 108, Server) already exists.,

Event ID

f5b80724129648118365d6d77fc12f4f a835b2461196413eb103165b6204783e 9b6a8922e2fb414da0a76ad311d7f14e f89c9fddb1624476b6cb3aafaee03198

hubertdeng123 commented 1 year ago

Interesting, how often does this happen? Seems to somehow be related to key value errors?

AlexanderRydberg commented 1 year ago

Prior to opening this issue it happened alot. Several times a day when navigating in the ui. Now it has been a long holiday here in Sweden so that could explain why I havent gotten the issue as often. I suspect that the duplicated keys are a result of the latest bump in version where postgres got updated, but I can be wrong here.

AlexanderRydberg commented 1 year ago

After an upgrade to 23.5.0 (and now 23.5.1) the issue is still present but no error is reported to sentry :/

hubertdeng123 commented 1 year ago

Darn, would it be possible to try manually deleting the duplicate keys in your postgres db?

AlexanderRydberg commented 1 year ago

Missed  your last comment! I dont think those duplicated keys are related to my issue since I have the same errors in our test-env that doesnt get these snuba-errors. Wrote some comments here: #2171 Im a bit too scared to do any commands in postgres :laughing:

AlexanderRydberg commented 1 year ago

I have a theory on when it happens. Not really much to go on but most mornings it will time out. I guess it happens when no one has navigated to the gui for a good while. Could that make sense?