Closed alekseyp closed 1 year ago
Is this restore being done on a clean install of sentry?
@hubertdeng123 yes it is.
Here is the full log.
root@sentry:/opt/sentry# git log -2
commit 2badb5f42ac804b0083c0d4c84fbbbf1f4d1ab8e (HEAD, tag: 23.3.1)
Author: getsentry-bot <bot@sentry.io>
Date: Wed Mar 22 18:27:42 2023 +0000
release: 23.3.1
commit 0531f2a6a6011a67f35d8cb9d05c235b7679be08
Author: Chad Whitacre <chadwhitacre@sentry.io>
Date: Tue Mar 21 15:25:48 2023 -0700
Bump Kafka version to keep up with SaaS (#2037)
root@sentry:/opt/sentry# git status
HEAD detached at 23.3.1
Untracked files:
(use "git add <file>..." to include in what will be committed)
sentry/backup.json
nothing added to commit but untracked files present (use "git add" to track)
root@sentry:/opt/sentry# docker compose down -v
[+] Running 24/24
✔ Container sentry-self-hosted-symbolicator-1 Removed 0.0s
✔ Container sentry-self-hosted-postgres-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1 Removed 0.0s
✔ Container sentry-self-hosted-smtp-1 Remo... 0.0s
✔ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Removed 0.0s
✔ Container sentry-self-hosted-memcached-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-api-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-sessions-consumer-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-transactions-consumer-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-consumer-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-replacer-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-outcomes-consumer-1 Removed 0.0s
✔ Container sentry-self-hosted-kafka-1 Rem... 0.0s
✔ Container sentry-self-hosted-redis-1 Rem... 0.0s
✔ Container sentry-self-hosted-clickhouse-1 Removed 0.0s
✔ Container sentry-self-hosted-zookeeper-1 Removed 0.0s
✔ Volume sentry-self-hosted_sentry-nginx-cache Removed 0.0s
✔ Volume sentry-self-hosted_sentry-kafka-log Removed 0.0s
✔ Volume sentry-self-hosted_sentry-smtp-log Removed 0.0s
✔ Volume sentry-self-hosted_sentry-zookeeper-log Removed 0.0s
✔ Volume sentry-self-hosted_sentry-clickhouse-log Removed 0.0s
✔ Volume sentry-self-hosted_sentry-secrets Removed 0.0s
✔ Volume sentry-self-hosted_sentry-smtp Re... 0.0s
✔ Network sentry-self-hosted_default Remov... 0.2s
root@sentry:/opt/sentry# ./scripts/restore.sh
▶ Initializing Docker Compose ...
▶ Detecting Docker platform
Detected Docker platform is linux/amd64
▶ Setting up error handling ...
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 292B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/debian:bullseye-slim
#3 DONE 0.1s
#4 [1/2] FROM docker.io/library/debian:bullseye-slim@sha256:9404b05bd09b57c76eccc0c5505b3c88b5feccac808d9b193a4fbac87bb44745
#4 DONE 0.0s
#5 [2/2] RUN set -x && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends jq && apt-get clean && rm -rf /var/lib/apt/lists/*
#5 CACHED
#6 exporting to image
#6 exporting layers done
#6 writing image sha256:2d5f23c468fd17ea27d0dd4c2984751e8206e0d4f3c55817166edc54d3b820ca done
#6 naming to docker.io/library/sentry-self-hosted-jq-local done
#6 DONE 0.0s
Hey, so ... we would love to automatically find out about issues with your
Sentry instance so that we can improve the product. Turns out there is an app
for that, called Sentry. Would you be willing to let us automatically send data
about your instance upstream to Sentry for development and debugging purposes?
y / yes / 1
n / no / 0
(Btw, we send this to our own self-hosted Sentry instance, not to Sentry SaaS,
so that we can be in this together.)
Here's the info we may collect:
- OS username
- IP address
- install log
- runtime errors
- performance data
Thirty (30) day retention. No marketing. Privacy policy at sentry.io/privacy.
y or n? y
Thank you. To avoid this prompt in the future, use one of these flags:
--report-self-hosted-issues
--no-report-self-hosted-issues
or set the REPORT_SELF_HOSTED_ISSUES environment variable:
REPORT_SELF_HOSTED_ISSUES=1 to send data
REPORT_SELF_HOSTED_ISSUES=0 to not send data
latest: Pulling from getsentry/sentry-cli
Digest: sha256:bf24b7dfc6ca1b84845e53cc7fbe04a5ee5a8df7319460a7cfc0a20736b420c9
Status: Image is up to date for getsentry/sentry-cli:latest
docker.io/getsentry/sentry-cli:latest
Network sentry-self-hosted_default Creating
Network sentry-self-hosted_default Created
Volume "sentry-self-hosted_sentry-nginx-cache" Creating
Volume "sentry-self-hosted_sentry-nginx-cache" Created
Volume "sentry-self-hosted_sentry-kafka-log" Creating
Volume "sentry-self-hosted_sentry-kafka-log" Created
Volume "sentry-self-hosted_sentry-smtp" Creating
Volume "sentry-self-hosted_sentry-smtp" Created
Volume "sentry-self-hosted_sentry-clickhouse-log" Creating
Volume "sentry-self-hosted_sentry-clickhouse-log" Created
Volume "sentry-self-hosted_sentry-zookeeper-log" Creating
Volume "sentry-self-hosted_sentry-zookeeper-log" Created
Volume "sentry-self-hosted_sentry-smtp-log" Creating
Volume "sentry-self-hosted_sentry-smtp-log" Created
Volume "sentry-self-hosted_sentry-secrets" Creating
Volume "sentry-self-hosted_sentry-secrets" Created
Container sentry-self-hosted-postgres-1 Creating
Container sentry-self-hosted-zookeeper-1 Creating
Container sentry-self-hosted-redis-1 Creating
Container sentry-self-hosted-symbolicator-1 Creating
Container sentry-self-hosted-memcached-1 Creating
Container sentry-self-hosted-smtp-1 Creating
Container sentry-self-hosted-clickhouse-1 Creating
Container sentry-self-hosted-zookeeper-1 Created
Container sentry-self-hosted-kafka-1 Creating
Container sentry-self-hosted-memcached-1 Created
Container sentry-self-hosted-smtp-1 Created
Container sentry-self-hosted-postgres-1 Created
Container sentry-self-hosted-redis-1 Created
Container sentry-self-hosted-clickhouse-1 Created
Container sentry-self-hosted-symbolicator-1 Created
Container sentry-self-hosted-kafka-1 Created
Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Creating
Container sentry-self-hosted-snuba-api-1 Creating
Container sentry-self-hosted-snuba-sessions-consumer-1 Creating
Container sentry-self-hosted-snuba-consumer-1 Creating
Container sentry-self-hosted-snuba-replacer-1 Creating
Container sentry-self-hosted-snuba-subscription-consumer-events-1 Creating
Container sentry-self-hosted-snuba-outcomes-consumer-1 Creating
Container sentry-self-hosted-snuba-transactions-consumer-1 Creating
Container sentry-self-hosted-snuba-replacer-1 Created
Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Created
Container sentry-self-hosted-snuba-outcomes-consumer-1 Created
Container sentry-self-hosted-snuba-subscription-consumer-events-1 Created
Container sentry-self-hosted-snuba-sessions-consumer-1 Created
Container sentry-self-hosted-snuba-consumer-1 Created
Container sentry-self-hosted-snuba-transactions-consumer-1 Created
Container sentry-self-hosted-snuba-api-1 Created
Container sentry-self-hosted-smtp-1 Starting
Container sentry-self-hosted-zookeeper-1 Starting
Container sentry-self-hosted-memcached-1 Starting
Container sentry-self-hosted-clickhouse-1 Starting
Container sentry-self-hosted-redis-1 Starting
Container sentry-self-hosted-symbolicator-1 Starting
Container sentry-self-hosted-postgres-1 Starting
Container sentry-self-hosted-postgres-1 Started
Container sentry-self-hosted-smtp-1 Started
Container sentry-self-hosted-clickhouse-1 Started
Container sentry-self-hosted-zookeeper-1 Started
Container sentry-self-hosted-zookeeper-1 Waiting
Container sentry-self-hosted-memcached-1 Started
Container sentry-self-hosted-redis-1 Started
Container sentry-self-hosted-symbolicator-1 Started
Container sentry-self-hosted-zookeeper-1 Healthy
Container sentry-self-hosted-kafka-1 Starting
Container sentry-self-hosted-kafka-1 Started
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-subscription-consumer-events-1 Starting
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Starting
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-api-1 Starting
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-sessions-consumer-1 Starting
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-outcomes-consumer-1 Starting
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-transactions-consumer-1 Starting
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-replacer-1 Starting
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-snuba-consumer-1 Starting
Container sentry-self-hosted-snuba-sessions-consumer-1 Started
Container sentry-self-hosted-snuba-subscription-consumer-events-1 Started
Container sentry-self-hosted-snuba-outcomes-consumer-1 Started
Container sentry-self-hosted-snuba-transactions-consumer-1 Started
Container sentry-self-hosted-snuba-consumer-1 Started
Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Started
Container sentry-self-hosted-snuba-replacer-1 Started
Container sentry-self-hosted-snuba-api-1 Started
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
21:44:44 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
21:44:45 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
21:45:11 [INFO] sentry.plugins.github: apps-not-configured
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
if key_extra_len is 0:
>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"
DETAIL: Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
raise_the_exception(self.db, e)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
raise exc_info[0](msg).with_traceback(exc_info[2])
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: UniqueViolation('duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"\nDETAIL: Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.\n')
SQL: INSERT INTO "sentry_useremail" ("id", "user_id", "email", "validation_hash", "date_hash_added", "is_verified") VALUES (%s, %s, %s, %s, %s, %s)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/sentry", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main
func(**kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
return ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 34, in import_
raise (e)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 23, in import_
obj.save()
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save
models.Model.save_base(self.object, using=using, raw=True, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
updated = self._save_table(
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 873, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 910, in _do_insert
return manager._insert([self], fields=fields, return_id=update_pk,
File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 582, in execute
return real_execute(self, sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
raise_the_exception(self.db, e)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
raise exc_info[0](msg).with_traceback(exc_info[2])
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq"\nDETAIL: Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.\n')
SQL: INSERT INTO "sentry_useremail" ("id", "user_id", "email", "validation_hash", "date_hash_added", "is_verified") VALUES (%s, %s, %s, %s, %s, %s)
Error in scripts/_lib.sh:70.
'docker-compose run --rm -T web import /etc/sentry/backup.json' exited with status 1
-> ./scripts/restore.sh:main:4
--> scripts/_lib.sh:restore:70
Looks like you've already sent this error to us, we're on it :)
To clarify, did you run
./scripts/backup.sh
docker compose down -v
on the same machine as
./scripts/restore.sh
?
The errors that are being displayed lead me to believe that there is already data in your postgres database, particularly this one 🤔. Restoring on an non-empty postgres database is not supported if that's what you are trying to do
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_useremail_user_id_email_ade975f1_uniq" DETAIL: Key (user_id, email)=(1, xxxxxx@yyyyyy.com) already exists.
Yes, I've tried to run it on the same machine and I also tried to run it on a brand-new machine, running the same version.
before each restore I always execute docker compose down -v
When restoring on the same server didn't work - I tried to run it on another server by copying the folder over. When that didn't work - I tried cloning the same version from scratch and only copying backup.json
There is something wrong with the backup file itself because it works on a brand-new setup. with just one username created via ./install.sh
script.
Got it, in order for me to take a look and investigate this, I'd need some more info on what might be in your backup file that could be causing this so I can replicate. Is there any information that you can give me that could be helpful?
I was able to restore the backup after deleting quite a few blocks.
I kept removing blocks that it was complaining about, like sentry.mail
, sentry.usermail
.
Maybe it's because we have multiple emails attached to the user?
Once I did those - I got one more error:
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_projectoptions_project_id_key_3c517bcf_uniq"\nDETAIL: Key (project_id, key)=(1, sentry:origins) already exists.\n')
SQL: UPDATE "sentry_projectoptions" SET "project_id" = %s, "key" = %s, "value" = %s WHERE "sentry_projectoptions"."id" = %s
And it's only one place where sentry:origins
is mentioned.
{
"model": "sentry.projectoption",
"pk": 3,
"fields": {
"project": 1,
"key": "sentry:origins",
"value": "[\"*\"]"
}
},
After deleting that one as well - backup got restored. From what it looks like - restore process is populating sentry.mail and sentry.usermail, from both sentry.user as well as sentry.mail and sentry.usermail (if that makes sense)
I've copied the backup file locally and getting different errors for the same backup file and same version.
It has the same md5 as the one on production server.
~/T/sentry ((23.3.1)|✔) $ md5sum sentry/backup.json
4806f32efc6ff3f04dc3c71328cd41f5 sentry/backup.json
~/T/sentry ((23.3.1)|✔) $ docker compose down -v
[+] Running 24/24
✔ Container sentry-self-hosted-snuba-consumer-1 Removed 0.1s
✔ Container sentry-self-hosted-symbolicator-1 Removed 0.1s
✔ Container sentry-self-hosted-postgres-1 Removed 0.1s
✔ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Removed 0.1s
✔ Container sentry-self-hosted-memcached-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-outcomes-consumer-1 Removed 0.0s
✔ Container sentry-self-hosted-smtp-1 Removed 0.1s
✔ Container sentry-self-hosted-snuba-replacer-1 Removed 0.1s
✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1 Removed 0.0s
✔ Container sentry-self-hosted-snuba-sessions-consumer-1 Removed 0.1s
✔ Container sentry-self-hosted-snuba-api-1 Removed 0.1s
✔ Container sentry-self-hosted-snuba-transactions-consumer-1 Removed 0.1s
✔ Container sentry-self-hosted-clickhouse-1 Removed 0.0s
✔ Container sentry-self-hosted-kafka-1 Removed 0.0s
✔ Container sentry-self-hosted-redis-1 Removed 0.0s
✔ Container sentry-self-hosted-zookeeper-1 Removed 0.0s
✔ Volume sentry-self-hosted_sentry-clickhouse-log Removed 0.0s
✔ Volume sentry-self-hosted_sentry-nginx-cache Removed 0.0s
✔ Volume sentry-self-hosted_sentry-kafka-log Removed 0.0s
✔ Volume sentry-self-hosted_sentry-secrets Removed 0.0s
✔ Volume sentry-self-hosted_sentry-smtp Removed 0.0s
✔ Volume sentry-self-hosted_sentry-zookeeper-log Removed 0.0s
✔ Volume sentry-self-hosted_sentry-smtp-log Removed 0.0s
✔ Network sentry-self-hosted_default Removed 0.1s
~/T/sentry ((23.3.1)|✔) $ ./scripts/restore.sh
▶ Initializing Docker Compose ...
▶ Detecting Docker platform
Detected Docker platform is linux/amd64
...
>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_option_key_key"
DETAIL: Key (key)=(auth.allow-registration) already exists.
...
grep allow-registration sentry/backup.json
"key": "auth.allow-registration",
If I remove that block another would show up as a duplicate.
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_option_key_key"\nDETAIL: Key (key)=(mail.port) already exists.\n')
SQL: UPDATE "sentry_option" SET "key" = %s, "value" = %s, "last_updated" = %s WHERE "sentry_option"."id" = %s
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "sentry_option_key_key"\nDETAIL: Key (key)=(mail.use-tls) already exists.\n')
SQL: UPDATE "sentry_option" SET "key" = %s, "value" = %s, "last_updated" = %s WHERE "sentry_option"."id" = %s
I don't understand why the same backup, running on the same version would provide different errors :/
Yeah, this is really helpful information, thanks for sharing. Yeah, I'm not quite sure why the same backup for you is providing you different errors either on the same version. You're not getting anything related to the same email errors locally?
Alright - it think the problem is less complicated than I made it.
Turns out that docker compose down -v
doesn't delete named sentry-posgres
volume.
So you were right about "not being a clean install".
I still get an error, but only one.
Here is the full re-tested flow.
docker compose down -v
docker volume rm sentry-postgres sentry-data sentry-clickhouse sentry-zookeeper sentry-redis sentry-symbolicator
./install.sh --skip-user-creation
./scripts/restore.sh
At that stage it would fail with
>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_projectoptions_project_id_key_3c517bcf_uniq"
DETAIL: Key (project_id, key)=(1, sentry:origins) already exists.
To fix it I delete the next block from backup.json
{
"model": "sentry.projectoption",
"pk": 3,
"fields": {
"project": 1,
"key": "sentry:origins",
"value": "[\"*\"]"
}
},
and repeat the steps above by deleting all volumes, installing and restoring again.
That way everything is restored. Including all emails, etc
One more thing to note, after restoring the backup, our GitLab integration didn't restore correctly. It has created custom (internal) integration instead and we had to redo the original one manually. If it's a bug - and not expected behaviour - I can create new issues.
This issue has gone three weeks without activity. In another week, I will close it.
But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog
or Status: In Progress
, I will leave it alone ... forever!
"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀
Self-Hosted Version
23.3.1
CPU Architecture
x86_64
Docker Version
23.0.3
Docker Compose Version
2.17.2
Steps to Reproduce
./scripts/backup.sh docker compose down -v ./scripts/restore.sh
Expected Result
Successful restore
Actual Result
Event ID
sentry-envelope-b56300705b3e0b527eb6f385eb213291