search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.9k stars 1.77k forks source link

Restore fails with UniqueViolation: duplicate key value violates unique constraint #2081

Closed alekseyp closed 1 year ago

alekseyp commented 1 year ago

Self-Hosted Version

23.3.1

CPU Architecture

x86_64

Docker Version

23.0.3

Docker Compose Version

2.17.2

Steps to Reproduce

./scripts/backup.sh docker compose down -v ./scripts/restore.sh

Expected Result

Successful restore

Actual Result

21:04:45 [INFO] sentry.plugins.github: apps-not-configured
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key_extra_len is 0:
>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"
DETAIL:  Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: UniqueViolation('duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"\nDETAIL:  Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.\n')
SQL: INSERT INTO "sentry_useremail" ("id", "user_id", "email", "validation_hash", "date_hash_added", "is_verified") VALUES (%s, %s, %s, %s, %s, %s)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 34, in import_
    raise (e)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 23, in import_
    obj.save()
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 910, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 582, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"\nDETAIL:  Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.\n')
SQL: INSERT INTO "sentry_useremail" ("id", "user_id", "email", "validation_hash", "date_hash_added", "is_verified") VALUES (%s, %s, %s, %s, %s, %s)
Error in scripts/_lib.sh:70.
'docker-compose run --rm -T web import /etc/sentry/backup.json' exited with status 1
-> ./scripts/restore.sh:main:4
--> scripts/_lib.sh:restore:70

Event ID

sentry-envelope-b56300705b3e0b527eb6f385eb213291

hubertdeng123 commented 1 year ago

Is this restore being done on a clean install of sentry?

alekseyp commented 1 year ago

@hubertdeng123 yes it is.

Here is the full log.

root@sentry:/opt/sentry# git log -2
commit 2badb5f42ac804b0083c0d4c84fbbbf1f4d1ab8e (HEAD, tag: 23.3.1)
Author: getsentry-bot <bot@sentry.io>
Date:   Wed Mar 22 18:27:42 2023 +0000

    release: 23.3.1

commit 0531f2a6a6011a67f35d8cb9d05c235b7679be08
Author: Chad Whitacre <chadwhitacre@sentry.io>
Date:   Tue Mar 21 15:25:48 2023 -0700

    Bump Kafka version to keep up with SaaS (#2037)
root@sentry:/opt/sentry# git status
HEAD detached at 23.3.1
Untracked files:
  (use "git add <file>..." to include in what will be committed)
    sentry/backup.json

nothing added to commit but untracked files present (use "git add" to track)
root@sentry:/opt/sentry# docker compose down -v
[+] Running 24/24
 ✔ Container sentry-self-hosted-symbolicator-1                              Removed                                       0.0s
 ✔ Container sentry-self-hosted-postgres-1                                  Removed                                       0.0s
 ✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1        Removed                                       0.0s
 ✔ Container sentry-self-hosted-smtp-1                                      Remo...                                       0.0s
 ✔ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Removed                                       0.0s
 ✔ Container sentry-self-hosted-memcached-1                                 Removed                                       0.0s
 ✔ Container sentry-self-hosted-snuba-api-1                                 Removed                                       0.0s
 ✔ Container sentry-self-hosted-snuba-sessions-consumer-1                   Removed                                       0.0s
 ✔ Container sentry-self-hosted-snuba-transactions-consumer-1               Removed                                       0.0s
 ✔ Container sentry-self-hosted-snuba-consumer-1                            Removed                                       0.0s
 ✔ Container sentry-self-hosted-snuba-replacer-1                            Removed                                       0.0s
 ✔ Container sentry-self-hosted-snuba-outcomes-consumer-1                   Removed                                       0.0s
 ✔ Container sentry-self-hosted-kafka-1                                     Rem...                                        0.0s
 ✔ Container sentry-self-hosted-redis-1                                     Rem...                                        0.0s
 ✔ Container sentry-self-hosted-clickhouse-1                                Removed                                       0.0s
 ✔ Container sentry-self-hosted-zookeeper-1                                 Removed                                       0.0s
 ✔ Volume sentry-self-hosted_sentry-nginx-cache                             Removed                                       0.0s
 ✔ Volume sentry-self-hosted_sentry-kafka-log                               Removed                                       0.0s
 ✔ Volume sentry-self-hosted_sentry-smtp-log                                Removed                                       0.0s
 ✔ Volume sentry-self-hosted_sentry-zookeeper-log                           Removed                                       0.0s
 ✔ Volume sentry-self-hosted_sentry-clickhouse-log                          Removed                                       0.0s
 ✔ Volume sentry-self-hosted_sentry-secrets                                 Removed                                       0.0s
 ✔ Volume sentry-self-hosted_sentry-smtp                                    Re...                                         0.0s
 ✔ Network sentry-self-hosted_default                                       Remov...                                      0.2s
root@sentry:/opt/sentry# ./scripts/restore.sh
▶ Initializing Docker Compose ...

▶ Detecting Docker platform
Detected Docker platform is linux/amd64

▶ Setting up error handling ...
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 292B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/debian:bullseye-slim
#3 DONE 0.1s

#4 [1/2] FROM docker.io/library/debian:bullseye-slim@sha256:9404b05bd09b57c76eccc0c5505b3c88b5feccac808d9b193a4fbac87bb44745
#4 DONE 0.0s

#5 [2/2] RUN set -x   && apt-get update   && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends jq   && apt-get clean   && rm -rf /var/lib/apt/lists/*
#5 CACHED

#6 exporting to image
#6 exporting layers done
#6 writing image sha256:2d5f23c468fd17ea27d0dd4c2984751e8206e0d4f3c55817166edc54d3b820ca done
#6 naming to docker.io/library/sentry-self-hosted-jq-local done
#6 DONE 0.0s

Hey, so ... we would love to automatically find out about issues with your
Sentry instance so that we can improve the product. Turns out there is an app
for that, called Sentry. Would you be willing to let us automatically send data
about your instance upstream to Sentry for development and debugging purposes?

  y / yes / 1
  n / no / 0

(Btw, we send this to our own self-hosted Sentry instance, not to Sentry SaaS,
so that we can be in this together.)

Here's the info we may collect:

  - OS username
  - IP address
  - install log
  - runtime errors
  - performance data

Thirty (30) day retention. No marketing. Privacy policy at sentry.io/privacy.

y or n? y

Thank you. To avoid this prompt in the future, use one of these flags:

  --report-self-hosted-issues
  --no-report-self-hosted-issues

or set the REPORT_SELF_HOSTED_ISSUES environment variable:

  REPORT_SELF_HOSTED_ISSUES=1 to send data
  REPORT_SELF_HOSTED_ISSUES=0 to not send data

latest: Pulling from getsentry/sentry-cli
Digest: sha256:bf24b7dfc6ca1b84845e53cc7fbe04a5ee5a8df7319460a7cfc0a20736b420c9
Status: Image is up to date for getsentry/sentry-cli:latest
docker.io/getsentry/sentry-cli:latest

 Network sentry-self-hosted_default  Creating
 Network sentry-self-hosted_default  Created
 Volume "sentry-self-hosted_sentry-nginx-cache"  Creating
 Volume "sentry-self-hosted_sentry-nginx-cache"  Created
 Volume "sentry-self-hosted_sentry-kafka-log"  Creating
 Volume "sentry-self-hosted_sentry-kafka-log"  Created
 Volume "sentry-self-hosted_sentry-smtp"  Creating
 Volume "sentry-self-hosted_sentry-smtp"  Created
 Volume "sentry-self-hosted_sentry-clickhouse-log"  Creating
 Volume "sentry-self-hosted_sentry-clickhouse-log"  Created
 Volume "sentry-self-hosted_sentry-zookeeper-log"  Creating
 Volume "sentry-self-hosted_sentry-zookeeper-log"  Created
 Volume "sentry-self-hosted_sentry-smtp-log"  Creating
 Volume "sentry-self-hosted_sentry-smtp-log"  Created
 Volume "sentry-self-hosted_sentry-secrets"  Creating
 Volume "sentry-self-hosted_sentry-secrets"  Created
 Container sentry-self-hosted-postgres-1  Creating
 Container sentry-self-hosted-zookeeper-1  Creating
 Container sentry-self-hosted-redis-1  Creating
 Container sentry-self-hosted-symbolicator-1  Creating
 Container sentry-self-hosted-memcached-1  Creating
 Container sentry-self-hosted-smtp-1  Creating
 Container sentry-self-hosted-clickhouse-1  Creating
 Container sentry-self-hosted-zookeeper-1  Created
 Container sentry-self-hosted-kafka-1  Creating
 Container sentry-self-hosted-memcached-1  Created
 Container sentry-self-hosted-smtp-1  Created
 Container sentry-self-hosted-postgres-1  Created
 Container sentry-self-hosted-redis-1  Created
 Container sentry-self-hosted-clickhouse-1  Created
 Container sentry-self-hosted-symbolicator-1  Created
 Container sentry-self-hosted-kafka-1  Created
 Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Creating
 Container sentry-self-hosted-snuba-api-1  Creating
 Container sentry-self-hosted-snuba-sessions-consumer-1  Creating
 Container sentry-self-hosted-snuba-consumer-1  Creating
 Container sentry-self-hosted-snuba-replacer-1  Creating
 Container sentry-self-hosted-snuba-subscription-consumer-events-1  Creating
 Container sentry-self-hosted-snuba-outcomes-consumer-1  Creating
 Container sentry-self-hosted-snuba-transactions-consumer-1  Creating
 Container sentry-self-hosted-snuba-replacer-1  Created
 Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Created
 Container sentry-self-hosted-snuba-outcomes-consumer-1  Created
 Container sentry-self-hosted-snuba-subscription-consumer-events-1  Created
 Container sentry-self-hosted-snuba-sessions-consumer-1  Created
 Container sentry-self-hosted-snuba-consumer-1  Created
 Container sentry-self-hosted-snuba-transactions-consumer-1  Created
 Container sentry-self-hosted-snuba-api-1  Created
 Container sentry-self-hosted-smtp-1  Starting
 Container sentry-self-hosted-zookeeper-1  Starting
 Container sentry-self-hosted-memcached-1  Starting
 Container sentry-self-hosted-clickhouse-1  Starting
 Container sentry-self-hosted-redis-1  Starting
 Container sentry-self-hosted-symbolicator-1  Starting
 Container sentry-self-hosted-postgres-1  Starting
 Container sentry-self-hosted-postgres-1  Started
 Container sentry-self-hosted-smtp-1  Started
 Container sentry-self-hosted-clickhouse-1  Started
 Container sentry-self-hosted-zookeeper-1  Started
 Container sentry-self-hosted-zookeeper-1  Waiting
 Container sentry-self-hosted-memcached-1  Started
 Container sentry-self-hosted-redis-1  Started
 Container sentry-self-hosted-symbolicator-1  Started
 Container sentry-self-hosted-zookeeper-1  Healthy
 Container sentry-self-hosted-kafka-1  Starting
 Container sentry-self-hosted-kafka-1  Started
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-subscription-consumer-events-1  Starting
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Starting
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-api-1  Starting
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-sessions-consumer-1  Starting
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-outcomes-consumer-1  Starting
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-transactions-consumer-1  Starting
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-replacer-1  Starting
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-consumer-1  Starting
 Container sentry-self-hosted-snuba-sessions-consumer-1  Started
 Container sentry-self-hosted-snuba-subscription-consumer-events-1  Started
 Container sentry-self-hosted-snuba-outcomes-consumer-1  Started
 Container sentry-self-hosted-snuba-transactions-consumer-1  Started
 Container sentry-self-hosted-snuba-consumer-1  Started
 Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Started
 Container sentry-self-hosted-snuba-replacer-1  Started
 Container sentry-self-hosted-snuba-api-1  Started
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
21:44:44 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
21:44:45 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
21:45:11 [INFO] sentry.plugins.github: apps-not-configured
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key_extra_len is 0:
>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"
DETAIL:  Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: UniqueViolation('duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"\nDETAIL:  Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.\n')
SQL: INSERT INTO "sentry_useremail" ("id", "user_id", "email", "validation_hash", "date_hash_added", "is_verified") VALUES (%s, %s, %s, %s, %s, %s)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 34, in import_
    raise (e)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 23, in import_
    obj.save()
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 910, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 582, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"\nDETAIL:  Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.\n')
SQL: INSERT INTO "sentry_useremail" ("id", "user_id", "email", "validation_hash", "date_hash_added", "is_verified") VALUES (%s, %s, %s, %s, %s, %s)
Error in scripts/_lib.sh:70.
'docker-compose run --rm -T web import /etc/sentry/backup.json' exited with status 1
-> ./scripts/restore.sh:main:4
--> scripts/_lib.sh:restore:70

Looks like you've already sent this error to us, we're on it :)
hubertdeng123 commented 1 year ago

To clarify, did you run

./scripts/backup.sh
docker compose down -v

on the same machine as 

./scripts/restore.sh

?

The errors that are being displayed lead me to believe that there is already data in your postgres database, particularly this one 🤔. Restoring on an non-empty postgres database is not supported if that's what you are trying to do psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq" DETAIL: Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.

alekseyp commented 1 year ago

Yes, I've tried to run it on the same machine and I also tried to run it on a brand-new machine, running the same version.

before each restore I always execute docker compose down -v

When restoring on the same server didn't work - I tried to run it on another server by copying the folder over. When that didn't work - I tried cloning the same version from scratch and only copying backup.json

There is something wrong with the backup file itself because it works on a brand-new setup. with just one username created via ./install.sh script.

hubertdeng123 commented 1 year ago

Got it, in order for me to take a look and investigate this, I'd need some more info on what might be in your backup file that could be causing this so I can replicate. Is there any information that you can give me that could be helpful?

alekseyp commented 1 year ago

I was able to restore the backup after deleting quite a few blocks.

I kept removing blocks that it was complaining about, like sentry.mail, sentry.usermail. Maybe it's because we have multiple emails attached to the user?

Once I did those - I got one more error:

django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_projectoptions_project_id_key_3c517bcf_uniq"\nDETAIL:  Key (project_id, key)=(1, sentry:origins) already exists.\n')
SQL: UPDATE "sentry_projectoptions" SET "project_id" = %s, "key" = %s, "value" = %s WHERE "sentry_projectoptions"."id" = %s

And it's only one place where sentry:origins is mentioned. 

{
  "model": "sentry.projectoption",
  "pk": 3,
  "fields": {
    "project": 1,
    "key": "sentry:origins",
    "value": "[\"*\"]"
  }
},

After deleting that one as well - backup got restored. From what it looks like - restore process is populating sentry.mail and sentry.usermail, from both sentry.user as well as sentry.mail and sentry.usermail (if that makes sense)

alekseyp commented 1 year ago

I've copied the backup file locally and getting different errors for the same backup file and same version.

It has the same md5 as the one on production server.

~/T/sentry ((23.3.1)|✔) $ md5sum sentry/backup.json
4806f32efc6ff3f04dc3c71328cd41f5  sentry/backup.json
~/T/sentry ((23.3.1)|✔) $ docker compose down -v
[+] Running 24/24
 ✔ Container sentry-self-hosted-snuba-consumer-1                            Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-symbolicator-1                              Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-postgres-1                                  Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-memcached-1                                 Removed                                                                                     0.0s
 ✔ Container sentry-self-hosted-snuba-outcomes-consumer-1                   Removed                                                                                     0.0s
 ✔ Container sentry-self-hosted-smtp-1                                      Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-snuba-replacer-1                            Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1        Removed                                                                                     0.0s
 ✔ Container sentry-self-hosted-snuba-sessions-consumer-1                   Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-snuba-api-1                                 Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-snuba-transactions-consumer-1               Removed                                                                                     0.1s
 ✔ Container sentry-self-hosted-clickhouse-1                                Removed                                                                                     0.0s
 ✔ Container sentry-self-hosted-kafka-1                                     Removed                                                                                     0.0s
 ✔ Container sentry-self-hosted-redis-1                                     Removed                                                                                     0.0s
 ✔ Container sentry-self-hosted-zookeeper-1                                 Removed                                                                                     0.0s
 ✔ Volume sentry-self-hosted_sentry-clickhouse-log                          Removed                                                                                     0.0s
 ✔ Volume sentry-self-hosted_sentry-nginx-cache                             Removed                                                                                     0.0s
 ✔ Volume sentry-self-hosted_sentry-kafka-log                               Removed                                                                                     0.0s
 ✔ Volume sentry-self-hosted_sentry-secrets                                 Removed                                                                                     0.0s
 ✔ Volume sentry-self-hosted_sentry-smtp                                    Removed                                                                                     0.0s
 ✔ Volume sentry-self-hosted_sentry-zookeeper-log                           Removed                                                                                     0.0s
 ✔ Volume sentry-self-hosted_sentry-smtp-log                                Removed                                                                                     0.0s
 ✔ Network sentry-self-hosted_default                                       Removed                                                                                     0.1s
~/T/sentry ((23.3.1)|✔) $ ./scripts/restore.sh
▶ Initializing Docker Compose ...

▶ Detecting Docker platform
Detected Docker platform is linux/amd64
...
>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_option_key_key"
DETAIL:  Key (key)=(auth.allow-registration) already exists.
...
grep allow-registration sentry/backup.json
    "key": "auth.allow-registration",

If I remove that block another would show up as a duplicate.

django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_option_key_key"\nDETAIL:  Key (key)=(mail.port) already exists.\n')
SQL: UPDATE "sentry_option" SET "key" = %s, "value" = %s, "last_updated" = %s WHERE "sentry_option"."id" = %s
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_option_key_key"\nDETAIL:  Key (key)=(mail.use-tls) already exists.\n')
SQL: UPDATE "sentry_option" SET "key" = %s, "value" = %s, "last_updated" = %s WHERE "sentry_option"."id" = %s

I don't understand why the same backup, running on the same version would provide different errors :/

hubertdeng123 commented 1 year ago

Yeah, this is really helpful information, thanks for sharing. Yeah, I'm not quite sure why the same backup for you is providing you different errors either on the same version. You're not getting anything related to the same email errors locally?

alekseyp commented 1 year ago

Alright - it think the problem is less complicated than I made it.

Turns out that docker compose down -v doesn't delete named sentry-posgres volume. So you were right about "not being a clean install".

I still get an error, but only one.

Here is the full re-tested flow.

docker compose down -v
docker volume rm sentry-postgres sentry-data sentry-clickhouse sentry-zookeeper sentry-redis sentry-symbolicator
./install.sh --skip-user-creation
./scripts/restore.sh

At that stage it would fail with 

>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_projectoptions_project_id_key_3c517bcf_uniq"
DETAIL:  Key (project_id, key)=(1, sentry:origins) already exists.

To fix it I delete the next block from backup.json

{
  "model": "sentry.projectoption",
  "pk": 3,
  "fields": {
    "project": 1,
    "key": "sentry:origins",
    "value": "[\"*\"]"
  }
},

and repeat the steps above by deleting all volumes, installing and restoring again.

That way everything is restored. Including all emails, etc

alekseyp commented 1 year ago

One more thing to note, after restoring the backup, our GitLab integration didn't restore correctly. It has created custom (internal) integration instead and we had to redo the original one manually. If it's a bug - and not expected behaviour - I can create new issues.

Screenshot 2023-04-14 at 2 07 38 AM

github-actions[bot] commented 1 year ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀