search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.9k stars 1.78k forks source link

Upgrade to 23.5.2 failed #2180

Closed NovakHonza closed 1 year ago

NovakHonza commented 1 year ago

Self-Hosted Version

23.5.2

CPU Architecture

x86_64

Docker Version

24.0.2

Docker Compose Version

2.18.1

Steps to Reproduce

I tried to upgrade from 23.5.1 to 23.5.2. I didnt do anything special, just git pull & ./install.sh

Expected Result

I expected upgrade to the 23.5.2 version without any problem

Actual Result

▶ Parsing command line ...

▶ Detecting Docker platform Detected Docker platform is linux/amd64

▶ Initializing Docker Compose ...

▶ Setting up error handling ...

1 [internal] load .dockerignore

1 transferring context: 2B done

1 ...

2 [internal] load build definition from Dockerfile

2 transferring dockerfile: 292B done

2 DONE 0.2s

1 [internal] load .dockerignore

1 DONE 0.4s

3 [internal] load metadata for docker.io/library/debian:bullseye-slim

3 DONE 0.0s

4 [1/2] FROM docker.io/library/debian:bullseye-slim

4 DONE 0.0s

5 [2/2] RUN set -x && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends jq && apt-get clean && rm -rf /var/lib/apt/lists/*

5 CACHED

6 exporting to image

6 exporting layers done

6 writing image sha256:576d2de57338ba7d4820d62b13d2234e7c739da76bdaabfe6c137bdd4f6a35bf 0.0s done

6 naming to docker.io/library/sentry-self-hosted-jq-local 0.0s done

6 DONE 0.0s

Hey, so ... we would love to automatically find out about issues with your Sentry instance so that we can improve the product. Turns out there is an app for that, called Sentry. Would you be willing to let us automatically send data about your instance upstream to Sentry for development and debugging purposes?

y / yes / 1 n / no / 0

(Btw, we send this to our own self-hosted Sentry instance, not to Sentry SaaS, so that we can be in this together.)

Here's the info we may collect:

Thirty (30) day retention. No marketing. Privacy policy at sentry.io/privacy.

y or n? Understood. To avoid this prompt in the future, use one of these flags:

--report-self-hosted-issues --no-report-self-hosted-issues

or set the REPORT_SELF_HOSTED_ISSUES environment variable:

REPORT_SELF_HOSTED_ISSUES=1 to send data REPORT_SELF_HOSTED_ISSUES=0 to not send data

▶ Checking for latest commit ... skipped

▶ Checking minimum requirements ... Found Docker version 24.0.2 Found Docker Compose version 2.18.1

▶ Turning things off ... Image symbolicator-cleanup-self-hosted-local:latest Removing Image sentry-cleanup-self-hosted-local:latest Removing Image sentry-self-hosted-local:latest Removing Image clickhouse-self-hosted-local:latest Removing Image clickhouse-self-hosted-local:latest Removed Image symbolicator-cleanup-self-hosted-local:latest Removed Image sentry-self-hosted-local:latest Removed Image sentry-cleanup-self-hosted-local:latest Removed

▶ Creating volumes for persistent storage ... Created sentry-clickhouse. Created sentry-data. Created sentry-kafka. Created sentry-postgres. Created sentry-redis. Created sentry-symbolicator. Created sentry-zookeeper.

▶ Ensuring files from examples ... sentry/sentry.conf.py already exists, skipped creation. sentry/config.yml already exists, skipped creation. symbolicator/config.yml already exists, skipped creation.

▶ Ensuring Relay credentials ... relay/config.yml already exists, skipped creation. relay/credentials.json already exists, skipped creation.

▶ Generating secret key ...

▶ Fetching and updating Docker images ... latest: Pulling from getsentry/sentry Digest: sha256:cfd527a678444f613f0ce75f85a62167b4929fd31513cafca749c390de74c85d Status: Image is up to date for getsentry/sentry:latest docker.io/getsentry/sentry:latest

▶ Building and tagging Docker images ...

1 [web internal] load .dockerignore

1 transferring context: 2B done

1 DONE 0.3s

2 [web internal] load build definition from Dockerfile

2 transferring dockerfile: 463B done

2 DONE 0.3s

3 [web internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [web 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [web internal] load build context

5 transferring context: 332B done

5 DONE 0.2s

6 [web 2/4] COPY . /usr/src/sentry

6 CACHED

7 [web 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [web 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [web] exporting to image

9 exporting layers done

9 writing image sha256:62099951cf0497be114faef6480ec6f0223a8fc2a5a19a361528df86b4b31629 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local 0.0s done

9 DONE 0.1s

1 [clickhouse internal] load .dockerignore

1 transferring context: 2B done

1 DONE 0.3s

2 [clickhouse internal] load build definition from Dockerfile

2 transferring dockerfile: 71B done

2 DONE 0.4s

3 [clickhouse internal] load metadata for docker.io/yandex/clickhouse-server:20.3.9.70

3 DONE 0.0s

4 [clickhouse 1/1] FROM docker.io/yandex/clickhouse-server:20.3.9.70

4 CACHED

5 [clickhouse] exporting to image

5 exporting layers done

5 writing image sha256:9b71090fa5753d3248c5c623eb9bc29b73850deb567e7b64f997f40544a44a31 0.1s done

5 naming to docker.io/library/clickhouse-self-hosted-local 0.0s done

5 DONE 0.1s

1 [cron internal] load .dockerignore

1 transferring context: 2B done

1 DONE 0.2s

2 [cron internal] load build definition from Dockerfile

2 transferring dockerfile: 463B done

2 DONE 0.3s

3 [cron internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [cron 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [cron internal] load build context

5 transferring context: 332B done

5 DONE 0.1s

6 [cron 2/4] COPY . /usr/src/sentry

6 CACHED

7 [cron 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [cron 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [cron] exporting to image

9 exporting layers done

9 writing image sha256:fefdfccc3f127be63e9ff5791854d178c71b0ff632a85c233a8b1d0dbcdffd4c 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

1 [ingest-consumer internal] load .dockerignore

1 transferring context: 2B done

1 DONE 0.2s

2 [ingest-consumer internal] load build definition from Dockerfile

2 transferring dockerfile: 463B done

2 DONE 0.4s

3 [ingest-consumer internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [ingest-consumer 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [ingest-consumer internal] load build context

5 transferring context: 332B done

5 DONE 0.4s

6 [ingest-consumer 2/4] COPY . /usr/src/sentry

6 CACHED

7 [ingest-consumer 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [ingest-consumer 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [ingest-consumer] exporting to image

9 exporting layers done

9 writing image sha256:a315819f7d8233862cd0c94a6430dc978d0ba1520e033d05efb2354d09527ce7 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

1 [ingest-replay-recordings internal] load build definition from Dockerfile

1 transferring dockerfile:

1 transferring dockerfile: 463B done

1 DONE 0.3s

2 [ingest-replay-recordings internal] load .dockerignore

2 transferring context: 2B done

2 DONE 0.4s

3 [ingest-replay-recordings internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [ingest-replay-recordings 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [ingest-replay-recordings internal] load build context

5 transferring context: 332B done

5 DONE 0.2s

6 [ingest-replay-recordings 2/4] COPY . /usr/src/sentry

6 CACHED

7 [ingest-replay-recordings 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [ingest-replay-recordings 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [ingest-replay-recordings] exporting to image

9 exporting layers done

9 writing image sha256:c25cd377a8dbcc6a8dea5b579c8f3d2b65ab2c6b66ea9f74c5a4dd9aa5525356 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

1 [web internal] load .dockerignore

1 transferring context: 2B done

1 DONE 0.2s

2 [web internal] load build definition from Dockerfile

2 transferring dockerfile: 463B done

2 DONE 0.3s

3 [web internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [web 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [web internal] load build context

5 transferring context: 332B done

5 DONE 0.1s

6 [web 2/4] COPY . /usr/src/sentry

6 CACHED

7 [web 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [web 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [web] exporting to image

9 exporting layers done

9 writing image sha256:62099951cf0497be114faef6480ec6f0223a8fc2a5a19a361528df86b4b31629 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

1 [post-process-forwarder-errors internal] load build definition from Dockerfile

1 transferring dockerfile: 463B done

1 ...

2 [post-process-forwarder-errors internal] load .dockerignore

2 transferring context: 2B done

2 DONE 0.2s

1 [post-process-forwarder-errors internal] load build definition from Dockerfile

1 DONE 0.3s

3 [post-process-forwarder-errors internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [post-process-forwarder-errors 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [post-process-forwarder-errors internal] load build context

5 transferring context: 332B done

5 DONE 0.2s

6 [post-process-forwarder-errors 2/4] COPY . /usr/src/sentry

6 CACHED

7 [post-process-forwarder-errors 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [post-process-forwarder-errors 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [post-process-forwarder-errors] exporting to image

9 exporting layers done

9 writing image sha256:3bd4030788ce4d2350d05604919138b7f2a0b5faa48ddd2501e415b4e9ccafbd 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

1 [post-process-forwarder-transactions internal] load build definition from Dockerfile

1 transferring dockerfile: 463B done

1 DONE 0.2s

2 [post-process-forwarder-transactions internal] load .dockerignore

2 transferring context: 2B done

2 DONE 0.3s

3 [post-process-forwarder-transactions internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [post-process-forwarder-transactions 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [post-process-forwarder-transactions internal] load build context

5 transferring context: 332B done

5 DONE 0.1s

6 [post-process-forwarder-transactions 2/4] COPY . /usr/src/sentry

6 CACHED

7 [post-process-forwarder-transactions 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [post-process-forwarder-transactions 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [post-process-forwarder-transactions] exporting to image

9 exporting layers done

9 writing image sha256:1f040d2da180b8839171db57d56a6cfad8b99d0f4aed8c7b94bd95a2b33609ca 0.1s done

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

1 [sentry-cleanup internal] load build definition from Dockerfile

1 transferring dockerfile:

1 transferring dockerfile: 469B done

1 DONE 0.3s

2 [sentry-cleanup internal] load .dockerignore

2 transferring context: 2B done

2 DONE 0.4s

3 [sentry-cleanup internal] load metadata for docker.io/library/sentry-self-hosted-local:latest

3 DONE 0.0s

4 [sentry-cleanup 1/5] FROM docker.io/library/sentry-self-hosted-local

4 DONE 0.0s

5 [sentry-cleanup internal] load build context

5 transferring context: 35B done

5 DONE 0.2s

6 [sentry-cleanup 3/5] RUN if [ -z "${https_proxy}" ]; then echo "Acquire::https::proxy "${https_proxy}";" >> /etc/apt/apt.conf; fi

6 CACHED

7 [sentry-cleanup 4/5] RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*

7 CACHED

8 [sentry-cleanup 2/5] RUN if [ -z "${http_proxy}" ]; then echo "Acquire::http::proxy "${http_proxy}";" >> /etc/apt/apt.conf; fi

8 CACHED

9 [sentry-cleanup 5/5] COPY entrypoint.sh /entrypoint.sh

9 CACHED

10 [sentry-cleanup] exporting to image

10 exporting layers done

10 writing image sha256:562dc73854fa7a3034c29bc90038b3d48a909c66263471121292ecc3d312260c 0.0s done

10 naming to docker.io/library/sentry-cleanup-self-hosted-local 0.1s done

10 DONE 0.1s

1 [subscription-consumer-events internal] load .dockerignore

1 transferring context: 2B done

1 DONE 0.3s

2 [subscription-consumer-events internal] load build definition from Dockerfile

2 transferring dockerfile: 463B done

2 DONE 0.3s

3 [subscription-consumer-events internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [subscription-consumer-events 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [subscription-consumer-events internal] load build context

5 transferring context: 332B done

5 DONE 0.1s

6 [subscription-consumer-events 2/4] COPY . /usr/src/sentry

6 CACHED

7 [subscription-consumer-events 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [subscription-consumer-events 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [subscription-consumer-events] exporting to image

9 exporting layers done

9 writing image sha256:1f2730b7f61206325107753f9a142e12db624403a432eb15bdfed6694ec2636f 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

1 [subscription-consumer-transactions internal] load build definition from Dockerfile

1 transferring dockerfile: 463B done

1 DONE 0.2s

2 [subscription-consumer-transactions internal] load .dockerignore

2 transferring context: 2B done

2 DONE 0.4s

3 [subscription-consumer-transactions internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [subscription-consumer-transactions 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [subscription-consumer-transactions internal] load build context

5 transferring context: 332B done

5 DONE 0.2s

6 [subscription-consumer-transactions 2/4] COPY . /usr/src/sentry

6 CACHED

7 [subscription-consumer-transactions 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [subscription-consumer-transactions 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [subscription-consumer-transactions] exporting to image

9 exporting layers done

9 writing image sha256:0442cebae0a3a1e9c35fb4231793df67791c1dd02e06b1b1c0800ed2678c5fc2 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.2s

1 [symbolicator-cleanup internal] load build definition from Dockerfile

1 transferring dockerfile: 469B done

1 DONE 0.2s

2 [symbolicator-cleanup internal] load .dockerignore

2 transferring context: 2B done

2 DONE 0.3s

3 [symbolicator-cleanup internal] load metadata for docker.io/getsentry/symbolicator:latest

3 DONE 0.0s

4 [symbolicator-cleanup 1/5] FROM docker.io/getsentry/symbolicator:latest

4 DONE 0.0s

5 [symbolicator-cleanup internal] load build context

5 transferring context: 35B done

5 DONE 0.2s

6 [symbolicator-cleanup 2/5] RUN if [ -z "${http_proxy}" ]; then echo "Acquire::http::proxy "${http_proxy}";" >> /etc/apt/apt.conf; fi

6 CACHED

7 [symbolicator-cleanup 3/5] RUN if [ -z "${https_proxy}" ]; then echo "Acquire::https::proxy "${https_proxy}";" >> /etc/apt/apt.conf; fi

7 CACHED

8 [symbolicator-cleanup 4/5] RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*

8 CACHED

9 [symbolicator-cleanup 5/5] COPY entrypoint.sh /entrypoint.sh

9 CACHED

10 [symbolicator-cleanup] exporting to image

10 exporting layers done

10 writing image sha256:ec2f5ffecea1ce12bc4d6b06eb2b5895173b68c6430af25b6ca901ca17b8fcac 0.0s done

10 naming to docker.io/library/symbolicator-cleanup-self-hosted-local 0.1s done

10 DONE 0.1s

1 [worker internal] load build definition from Dockerfile

1 transferring dockerfile:

1 transferring dockerfile: 463B done

1 DONE 0.3s

2 [worker internal] load .dockerignore

2 transferring context: 2B done

2 DONE 0.3s

3 [worker internal] load metadata for docker.io/getsentry/sentry:latest

3 DONE 0.0s

4 [worker 1/4] FROM docker.io/getsentry/sentry:latest

4 DONE 0.0s

5 [worker internal] load build context

5 transferring context: 332B done

5 DONE 0.2s

6 [worker 2/4] COPY . /usr/src/sentry

6 CACHED

7 [worker 3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then /usr/src/sentry/enhance-image.sh; fi

7 CACHED

8 [worker 4/4] RUN if [ -s /usr/src/sentry/requirements.txt ]; then echo "sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image"; pip install -r /usr/src/sentry/requirements.txt; fi

8 CACHED

9 [worker] exporting to image

9 exporting layers done

9 writing image sha256:c351c8f7fd4c47ac7ed8f67e6c9e5a869a448826e2935764b05abd896a358545 0.0s done

9 naming to docker.io/library/sentry-self-hosted-local

9 naming to docker.io/library/sentry-self-hosted-local 0.1s done

9 DONE 0.1s

Docker images built.

▶ Downloading and installing wal2json ...

▶ Bootstrapping and migrating Snuba ... Network sentry_default Creating Network sentry_default Created Container sentry-zookeeper-1 Creating Container sentry-redis-1 Creating Container sentry-clickhouse-1 Creating Container sentry-clickhouse-1 Created Container sentry-redis-1 Created Container sentry-zookeeper-1 Created Container sentry-kafka-1 Creating Container sentry-kafka-1 Created Container sentry-redis-1 Starting Container sentry-clickhouse-1 Starting Container sentry-zookeeper-1 Starting Container sentry-clickhouse-1 Started Container sentry-redis-1 Started Container sentry-zookeeper-1 Started Container sentry-zookeeper-1 Waiting Container sentry-zookeeper-1 Healthy Container sentry-kafka-1 Starting Container sentry-kafka-1 Started 2023-05-31 21:15:22,027 Initializing Snuba... 2023-05-31 21:15:27,255 Snuba initialization took 5.230237970943563s 2023-05-31 21:15:27,920 Initializing Snuba... 2023-05-31 21:15:32,637 Snuba initialization took 4.719786162022501s 2023-05-31 21:15:32,639 Attempting to connect to Kafka (attempt 0)... 2023-05-31 21:15:32,707 Connected to Kafka on attempt 0 2023-05-31 21:15:32,709 Creating Kafka topics... Container sentry-redis-1 Running Container sentry-clickhouse-1 Running Container sentry-zookeeper-1 Running Container sentry-kafka-1 Running Container sentry-zookeeper-1 Waiting Container sentry-zookeeper-1 Healthy 2023-05-31 21:15:37,635 Initializing Snuba... 2023-05-31 21:15:42,286 Snuba initialization took 4.654038704931736s 2023-05-31 21:15:42,846 Initializing Snuba... 2023-05-31 21:15:47,011 Snuba initialization took 4.166968039935455s Finished running migrations

▶ Creating additional Kafka topics ...

▶ Ensuring proper PostgreSQL version ...

▶ Setting up / migrating database ... Container sentry-symbolicator-1 Creating Container sentry-smtp-1 Creating Container sentry-postgres-1 Creating Container sentry-memcached-1 Creating Container sentry-redis-1 Running Container sentry-zookeeper-1 Running Container sentry-clickhouse-1 Running Container sentry-kafka-1 Running Container sentry-snuba-subscription-consumer-transactions-1 Creating Container sentry-snuba-consumer-1 Creating Container sentry-snuba-subscription-consumer-events-1 Creating Container sentry-snuba-transactions-consumer-1 Creating Container sentry-snuba-api-1 Creating Container sentry-snuba-outcomes-consumer-1 Creating Container sentry-snuba-sessions-consumer-1 Creating Container sentry-snuba-replacer-1 Creating Container sentry-memcached-1 Created Container sentry-snuba-consumer-1 Created Container sentry-snuba-transactions-consumer-1 Created Container sentry-snuba-outcomes-consumer-1 Created Container sentry-symbolicator-1 Created Container sentry-postgres-1 Created Container sentry-snuba-replacer-1 Created Container sentry-snuba-sessions-consumer-1 Created Container sentry-snuba-subscription-consumer-transactions-1 Created Container sentry-smtp-1 Created Container sentry-snuba-api-1 Created Container sentry-snuba-subscription-consumer-events-1 Created Container sentry-symbolicator-1 Starting Container sentry-memcached-1 Starting Container sentry-postgres-1 Starting Container sentry-smtp-1 Starting Container sentry-zookeeper-1 Waiting Container sentry-zookeeper-1 Healthy Container sentry-redis-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-kafka-1 Waiting Container sentry-redis-1 Waiting Container sentry-kafka-1 Waiting Container sentry-kafka-1 Waiting Container sentry-redis-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-redis-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-kafka-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-redis-1 Waiting Container sentry-kafka-1 Waiting Container sentry-redis-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-redis-1 Waiting Container sentry-kafka-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-redis-1 Waiting Container sentry-kafka-1 Waiting Container sentry-kafka-1 Waiting Container sentry-clickhouse-1 Waiting Container sentry-clickhouse-1 Healthy Container sentry-redis-1 Healthy Container sentry-kafka-1 Healthy Container sentry-redis-1 Healthy Container sentry-redis-1 Healthy Container sentry-redis-1 Healthy Container sentry-redis-1 Healthy Container sentry-kafka-1 Healthy Container sentry-redis-1 Healthy Container sentry-clickhouse-1 Healthy Container sentry-kafka-1 Healthy Container sentry-clickhouse-1 Healthy Container sentry-snuba-subscription-consumer-transactions-1 Starting Container sentry-redis-1 Healthy Container sentry-snuba-consumer-1 Starting Container sentry-redis-1 Healthy Container sentry-snuba-api-1 Starting Container sentry-clickhouse-1 Healthy Container sentry-kafka-1 Healthy Container sentry-clickhouse-1 Healthy Container sentry-clickhouse-1 Healthy Container sentry-kafka-1 Healthy Container sentry-snuba-sessions-consumer-1 Starting Container sentry-clickhouse-1 Healthy Container sentry-kafka-1 Healthy Container sentry-kafka-1 Healthy Container sentry-snuba-transactions-consumer-1 Starting Container sentry-snuba-outcomes-consumer-1 Starting Container sentry-kafka-1 Healthy Container sentry-snuba-replacer-1 Starting Container sentry-clickhouse-1 Healthy Container sentry-snuba-subscription-consumer-events-1 Starting Container sentry-symbolicator-1 Started Container sentry-smtp-1 Started Container sentry-postgres-1 Started Container sentry-memcached-1 Started Container sentry-snuba-consumer-1 Started Container sentry-snuba-api-1 Started Container sentry-snuba-sessions-consumer-1 Started Container sentry-snuba-transactions-consumer-1 Started Container sentry-snuba-subscription-consumer-transactions-1 Started Container sentry-snuba-outcomes-consumer-1 Started Container sentry-snuba-replacer-1 Started Container sentry-snuba-subscription-consumer-events-1 Started Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://github.com/getsentry/self-hosted#enhance-sentry-image 21:16:41 [INFO] sentry.plugins.github: apps-not-configured Running migrations for default

Activitys: N/A% | | ETA: --:--:--

Activitys: N/A% | | ETA: --:--:--

Activitys: 1% | | ETA: 0:00:30

Activitys: 1% | | ETA: 0:00:23

Activitys: 2% |# | ETA: 0:00:19

Activitys: 3% |# | ETA: 0:00:16

Activitys: 4% |## | ETA: 0:00:14

Activitys: 5% |## | ETA: 0:00:13

Activitys: 6% |## | ETA: 0:00:14

Activitys: 7% |### | ETA: 0:00:13

Activitys: 7% |### | ETA: 0:00:13

Activitys: 8% |### | ETA: 0:00:13

Activitys: 8% |#### | ETA: 0:00:13

Activitys: 9% |#### | ETA: 0:00:13

Activitys: 10% |##### | ETA: 0:00:12

Activitys: 11% |##### | ETA: 0:00:12

Activitys: 12% |##### | ETA: 0:00:12

Activitys: 13% |###### | ETA: 0:00:11

Activitys: 14% |###### | ETA: 0:00:11

Activitys: 15% |###### | ETA: 0:00:11

Activitys: 16% |####### | ETA: 0:00:11

Activitys: 16% |####### | ETA: 0:00:11

Activitys: 17% |######## | ETA: 0:00:11

Activitys: 18% |######## | ETA: 0:00:10

Activitys: 19% |######### | ETA: 0:00:10

Activitys: 20% |######### | ETA: 0:00:10

Activitys: 21% |######### | ETA: 0:00:10

Activitys: 22% |########## | ETA: 0:00:09

Activitys: 23% |########## | ETA: 0:00:09

Activitys: 24% |########### | ETA: 0:00:09

Activitys: 25% |########### | ETA: 0:00:09

Activitys: 26% |############ | ETA: 0:00:09

Activitys: 27% |############ | ETA: 0:00:08

Activitys: 27% |############ | ETA: 0:00:08

Activitys: 28% |############# | ETA: 0:00:08

Activitys: 30% |############# | ETA: 0:00:08

Activitys: 31% |############## | ETA: 0:00:08

Activitys: 31% |############## | ETA: 0:00:08

Activitys: 32% |############### | ETA: 0:00:08

Activitys: 33% |############### | ETA: 0:00:07

Activitys: 34% |################ | ETA: 0:00:07

Activitys: 35% |################ | ETA: 0:00:07

Activitys: 36% |################ | ETA: 0:00:07

Activitys: 37% |################# | ETA: 0:00:07

Activitys: 38% |################# | ETA: 0:00:07

Activitys: 39% |################## | ETA: 0:00:06

Activitys: 40% |################## | ETA: 0:00:06

Activitys: 41% |################### | ETA: 0:00:06

Activitys: 42% |################### | ETA: 0:00:06

Activitys: 43% |################### | ETA: 0:00:06

Activitys: 43% |#################### | ETA: 0:00:06

Activitys: 45% |#################### | ETA: 0:00:06

Activitys: 45% |##################### | ETA: 0:00:06

Activitys: 46% |##################### | ETA: 0:00:06

Activitys: 47% |###################### | ETA: 0:00:05

Activitys: 48% |###################### | ETA: 0:00:05

Activitys: 49% |###################### | ETA: 0:00:05

Activitys: 50% |####################### | ETA: 0:00:05

Activitys: 50% |####################### | ETA: 0:00:05

Activitys: 51% |####################### | ETA: 0:00:05

Activitys: 52% |######################## | ETA: 0:00:05

Activitys: 53% |######################## | ETA: 0:00:05

Activitys: 54% |######################## | ETA: 0:00:05

Activitys: 55% |######################### | ETA: 0:00:05

Activitys: 55% |######################### | ETA: 0:00:05

Activitys: 56% |########################## | ETA: 0:00:05

Activitys: 57% |########################## | ETA: 0:00:04

Activitys: 58% |########################### | ETA: 0:00:04

Activitys: 59% |########################### | ETA: 0:00:04

Activitys: 60% |########################### | ETA: 0:00:04

Activitys: 61% |############################ | ETA: 0:00:04

Activitys: 62% |############################ | ETA: 0:00:04

Activitys: 62% |############################ | ETA: 0:00:04

Activitys: 63% |############################# | ETA: 0:00:04

Activitys: 64% |############################# | ETA: 0:00:04

Activitys: 65% |############################## | ETA: 0:00:04

Activitys: 65% |############################## | ETA: 0:00:03

Activitys: 66% |############################## | ETA: 0:00:03

Activitys: 67% |############################### | ETA: 0:00:03Operations to perform: Apply all migrations: admin, auth, contenttypes, nodestore, replays, sentry, sessions, sites, social_auth Running migrations: Applying sentry.0465_pickle_to_json_sentry_activity...Traceback (most recent call last): File "/usr/local/bin/sentry", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 193, in main func(kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner return ctx.invoke(f, *args, *kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 99, in upgrade _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore) File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 50, in _upgrade dj_call_command( File "/usr/local/lib/python3.8/site-packages/django/core/management/init.py", line 148, in call_command return command.execute(*args, *defaults) File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute output = self.handle(args, options) File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped res = handle_func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle post_migrate_state = executor.migrate( File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/monkey/executor.py", line 79, in apply_migration return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration state = migration.apply(state, schema_editor) File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/migrations.py", line 22, in apply return super().apply(project_state, schema_editor, collect_sql) File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply operation.database_forwards(self.app_label, schema_editor, old_state, project_state) File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards self.code(from_state.apps, schema_editor) File "/usr/local/lib/python3.8/site-packages/sentry/migrations/0465_pickle_to_json_sentry_activity.py", line 14, in _backfill obj.save(update_fields=["data"]) File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 743, in save self.save_base(using=using, force_insert=force_insert, File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 780, in save_base updated = self._save_table( File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 853, in _save_table updated = self._do_update(base_qs, using, pk_val, values, update_fields, File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 903, in _do_update return filtered._update(values) > 0 File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 760, in _update return query.get_compiler(self.db).execute_sql(CURSOR) File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql cursor = super().execute_sql(result_type) File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1129, in execute_sql sql, params = self.as_sql() File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1437, in as_sql val = field.get_db_prep_save(val, connection=self.connection) File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/init.py", line 793, in get_db_prep_save return self.get_db_prep_value(value, connection=connection, prepared=False) File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/init.py", line 788, in get_db_prep_value value = self.get_prep_value(value) File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/gzippeddict.py", line 79, in get_prep_value return json.dumps(value, default=jsonfield.default) File "/usr/local/lib/python3.8/site-packages/sentry/utils/json.py", line 110, in dumps return _default_encoder.encode(value) File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode return _iterencode(o, 0) File "/usr/local/lib/python3.8/site-packages/sentry/utils/json.py", line 57, in better_default_encoder raise TypeError(repr(o) + " is not JSON serializable") TypeError: <BaseQuerySet ['e3e420d427d18cb13e186da6b4f3f57b', '00f787beb49f06ec4fc79af85015d8c5', '970c79b40b668b3d172f3caba66c13ab', '4a92cc221936e936dc263b6138209dcb', '2070b0da9efb6666aa5dc73f5f75f77b', 'bb61c7ccf4eb6ca5ada21f6989dac222', '22b1b440161a6d0a6b011334e8c6f733', '89063020b7d43f574d127db4fc71937b', 'b348a7b9f2b48cf910be52bd2b796774', '792454165aad25b9f01b267ae5552830', 'e0aaea76a676d080696d47cd21afd46e', '13e91f868fc715f8ad58e6000abfa58f', 'a0f49dddafde7987158bf2a79ce45f23', 'bd7d67ab4a46f99c868e657f855dd229', '7c8cc715e5463063d5b8ba327aac3dbc', '5a2e0ab954955f8d161c6c2f99348517', 'c7e51967a40f1234f5e66fa0c6f4b263', '1f9f8410439b9502fd6433571f1c0d24', 'dd2b406b1aa151ce3040d140cda96683', 'cec86e6c78d894854eb78b3390be8649']> is not JSON serializable Error in install/set-up-and-migrate-database.sh:12. '$dcr web upgrade' exited with status 1 -> ./install.sh:main:34 --> install/set-up-and-migrate-database.sh:source:12

Cleaning up...

Event ID

No response

eduardopuente commented 1 year ago

same here, I was updating from 21.11.0:

File "/usr/local/lib/python3.8/site-packages/sentry/utils/json.py", line 57, in better_default_encoder
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <BaseQuerySet ['3ae73d1738edea7aa98055c104c3ad7a', '98d306c9997cb804982f601d7a7a56ef', 'dc1b2aec9e1456bc4ae58eb24dabf3d4', 'f25413e9352faea177ff50c52e693434', 'eff8d3c166ea0d1a7b0d298a4241a246', '3cf59d77a3a96f5dc8b404e50dc1ea23']> is not JSON serializable
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12
hubertdeng123 commented 1 year ago

Wonder if this has something to do with the pickle to json effort being made in getsentry/sentry. @eduardopuente which migration are you failing on here, is it 0465__pickle_to_json_sentry_activity too?

eduardopuente commented 1 year ago

@hubertdeng123 I was not able to identify the migration. However, I can confirm that upgrading to 23.5.1 worked fantastic

wampiedriessen commented 1 year ago

I have this exact same problem, also a BaseQuerySet ['1941c51769e492684f2083d372cc3763']> is not JSON serializable during the 0465__pickle_to_json_sentry_activity migration.

I was upgrading from 23.5.0 to 23.5.2

Changing the version to 23.5.1 worked for me

NovakHonza commented 1 year ago

I can confirm that rollback to 23.5.1 worked

tycho commented 1 year ago

I ran into something similar.

Running migrations for default
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, nodestore, replays, sentry, sessions, sites, social_auth
Running migrations:
Applying sentry.0468_pickle_to_json_sentry_rawevent...
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 99, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 50, in _upgrade
    dj_call_command(
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/monkey/executor.py", line 79, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/migrations/0468_pickle_to_json_sentry_rawevent.py", line 14, in _backfill
    obj.save(update_fields=["data"])
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 743, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 853, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 903, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1129, in execute_sql
    sql, params = self.as_sql()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1437, in as_sql
    val = field.get_db_prep_save(val, connection=self.connection)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 793, in get_db_prep_save
    return self.get_db_prep_value(value, connection=connection, prepared=False)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 788, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/node.py", line 242, in get_prep_value
    value.save()
  File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/node.py", line 156, in save
    nodestore.set_subkeys(self.id, subkeys)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 135, in <lambda>
    context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py", line 198, in inner
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/nodestore/base.py", line 244, in set_subkeys
    bytes_data = self._encode(data)
  File "/usr/local/lib/python3.8/site-packages/sentry/nodestore/base.py", line 204, in _encode
    lines = [json_dumps(data.pop(None)).encode("utf8")]
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 272, in default
    raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type NodeData is not JSON serializable
NovakHonza commented 1 year ago

Is it fixed in 23.6.x versions? can anyone test it, please?

mbudde commented 1 year ago

I got the Object of type NodeData is not JSON serializable error when trying to upgrade to 23.6.1. Upgrading to 23.5.1 instead worked.

wampiedriessen commented 1 year ago

Upgrading from 23.5.1 to 23.6.1 resolved the problem for me. No more migration errors during startup ¯\(ツ)/¯

marcodewaard commented 1 year ago

Probably the same issue over here.

Currently we are stuck at 23.5.1 because this is the last version that works. Every version after this one gives an error when running ./install.sh. I've tried probably every individual release after this one. The issue exists since the 23.5.2 release was installed and sentry_install_log-2023-06-06_14-59-43.txt states the following error: 

Running migrations for default
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key_extra_len is 0:
Raw Events: N/A% |                                             | ETA:  --:--:--Operations to perform:
  Apply all migrations: admin, auth, contenttypes, nodestore, replays, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0468_pickle_to_json_sentry_rawevent...Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 99, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 50, in _upgrade
    dj_call_command(
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/monkey/executor.py", line 79, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/migrations/0468_pickle_to_json_sentry_rawevent.py", line 14, in _backfill
    obj.save(update_fields=["data"])
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 743, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 853, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 903, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1129, in execute_sql
    sql, params = self.as_sql()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1437, in as_sql
    val = field.get_db_prep_save(val, connection=self.connection)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 793, in get_db_prep_save
    return self.get_db_prep_value(value, connection=connection, prepared=False)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 788, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/node.py", line 242, in get_prep_value
    value.save()
  File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/node.py", line 156, in save
    nodestore.set_subkeys(self.id, subkeys)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 135, in <lambda>
    context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py", line 198, in inner
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/nodestore/base.py", line 244, in set_subkeys
    bytes_data = self._encode(data)
  File "/usr/local/lib/python3.8/site-packages/sentry/nodestore/base.py", line 204, in _encode
    lines = [json_dumps(data.pop(None)).encode("utf8")]
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 272, in default
    raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type NodeData is not JSON serializable
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12
hubertdeng123 commented 1 year ago

Currently we are stuck at 23.5.1 because this is the last version that works. Every version after this one gives an error when running ./install.sh. I've tried probably every individual release after this one

Unfortunately, got nothing for you there. Perhaps data was corrupted during one of the migrations that were performed

hubertdeng123 commented 1 year ago

Were you able to try upgrading to 23.6.1? Looks like it was successful for someone else in the thread

marcodewaard commented 1 year ago

@hubertdeng123 Sadly no, did try almost every version between 23.5.0 and 23.8.0. All come back with the same error when running the install.sh script. 

  File "/usr/local/lib/python3.8/site-packages/sentry/migrations/0468_pickle_to_json_sentry_rawevent.py", line 14, in _backfill
    obj.save(update_fields=["data"])

So based on your response I guess rebuilding is going to be the only solution for us at this point.

id-sign commented 1 year ago

I have successfully upgraded to 23.7.x version and now I am on 23.8.0 without any problem

hubertdeng123 commented 1 year ago

Going to go ahead and close this, as a general solution seems to be in place