Open edgariscoding opened 1 month ago
@hubertdeng123 when doing research on this issue i noticed that you had done some work on UniqueViolation issues when restoring a backup, could you let me know if this is now fixed?
Connected to the postgres database to check what rows are conflicting in sentry_environment
... but it's empty. So it's complaining about a UniqueViolation even though the table is empty?
I've attempted removing all the sentry.environment
module nodes in the backup.json file, i still get a UniqueViolation duplicat key value for sentry_environment
Additionally, every time i re-run the import tool the organization_id keeps incrementing by 1.
reason: UniqueViolation(
'duplicate key value violates unique constraint "sentry_environment_organization_id_name_95a37dc7_uniq"\n
DETAIL: Key (organization_id, name)=(4, Production) already exists.\n')
SQL: INSERT INTO "sentry_environment" ("organization_id", "name", "date_added")
VALUES (%s, %s, %s) RETURNING "sentry_environment"."id"
)
Restore log:
sentryroot@CC-AZURE-SENTRY:~/sentry$ ./sentry-admin.sh import organizations backup.json
▶ Initializing Docker Compose ...
▶ Detecting Docker platform
Detected Docker platform is linux/amd64
Container sentry-self-hosted-redis-1 Running
Container sentry-self-hosted-smtp-1 Running
Container sentry-self-hosted-postgres-1 Running
Container sentry-self-hosted-memcached-1 Running
Container sentry-self-hosted-zookeeper-1 Running
Container sentry-self-hosted-symbolicator-1 Running
Container sentry-self-hosted-clickhouse-1 Running
Container sentry-self-hosted-kafka-1 Running
Container sentry-self-hosted-snuba-errors-consumer-1 Running
Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Running
Container sentry-self-hosted-snuba-transactions-consumer-1 Running
Container sentry-self-hosted-snuba-outcomes-billing-consumer-1 Running
Container sentry-self-hosted-snuba-outcomes-consumer-1 Running
Container sentry-self-hosted-snuba-api-1 Running
Container sentry-self-hosted-vroom-1 Running
Container sentry-self-hosted-snuba-replacer-1 Running
Container sentry-self-hosted-snuba-subscription-consumer-events-1 Running
Container sentry-self-hosted-zookeeper-1 Waiting
Container sentry-self-hosted-zookeeper-1 Healthy
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-kafka-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-redis-1 Waiting
Container sentry-self-hosted-clickhouse-1 Waiting
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-clickhouse-1 Healthy
Container sentry-self-hosted-redis-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Container sentry-self-hosted-kafka-1 Healthy
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Still importing: 5.00 seconds elapsed.
RpcImportError(
kind: IntegrityError,
on: InstanceID(model: 'sentry.environment'),
reason: UniqueViolation('duplicate key value violates unique constraint "sentry_environment_organization_id_name_95a37dc7_uniq"\nDETAIL: Key (organization_id, name)=(4, Production) already exists.\n')
SQL: INSERT INTO "sentry_environment" ("organization_id", "name", "date_added") VALUES (%s, %s, %s) RETURNING "sentry_environment"."id"
)
>> Are you restoring from a backup of the same version of Sentry?
>> Are you restoring onto a clean database?
>> If so then this IntegrityError might be our fault, you can open an issue here:
>> https://github.com/getsentry/sentry/issues/new/choose
RpcImportError(
kind: IntegrityError,
on: InstanceID(model: 'sentry.environment'),
reason: UniqueViolation('duplicate key value violates unique constraint "sentry_environment_organization_id_name_95a37dc7_uniq"\nDETAIL: Key (organization_id, name)=(4, Production) already exists.\n')
SQL: INSERT INTO "sentry_environment" ("organization_id", "name", "date_added") VALUES (%s, %s, %s) RETURNING "sentry_environment"."id"
)
Traceback (most recent call last):
File "/.venv/bin/sentry", line 4, in <module>
raise SystemExit(main())
^^^^^^
File "/usr/src/sentry/src/sentry/runner/main.py", line 148, in main
func(**kwargs)
File "/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/runner/decorators.py", line 35, in inner
return ctx.invoke(f, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/runner/commands/backup.py", line 764, in import_organizations
import_in_organization_scope(
File "/usr/src/sentry/src/sentry/backup/imports.py", line 541, in import_in_organization_scope
return _import(
^^^^^^^^
File "/usr/src/sentry/src/sentry/backup/imports.py", line 474, in _import
do_writes(pk_map)
File "/usr/src/sentry/src/sentry/backup/imports.py", line 403, in do_writes
do_write(import_write_context, pk_map, model_name, json_data, offset)
File "/usr/src/sentry/src/sentry/backup/imports.py", line 334, in do_write
raise ImportingError(result)
sentry.backup.imports.ImportingError: is_err=True kind='IntegrityError' on=InstanceID(model='sentry.environment', ordinal=None) left_pk=None right_pk=None reason='UniqueViolation(\'duplicate key value violates unique constraint "sentry_environment_organization_id_name_95a37dc7_uniq"\\nDETAIL: Key (organization_id, name)=(4, Production) already exists.\\n\')\nSQL: INSERT INTO "sentry_environment" ("organization_id", "name", "date_added") VALUES (%s, %s, %s) RETURNING "sentry_environment"."id"'
I was able to get sudo ./sentry-admin.sh import config backup.json
and sudo ./sentry-admin.sh import users backup.json
to work
but sudo ./sentry-admin.sh import organizations backup.json
fails with the same UniqueViolation error even if I remove all the sentry.environment
models from the backup.json file
More findings. after removing literally every model in the backup.json file... i kept getting the same error.
It's as if it was reading the backup.json file from somewhere else, so to test this theory I deleted the backup.json
file and re-ran the import command.
I expected for it to fail since the backup.json file is missing but it still gave the same UniqueViolation error.
I've made sure there's no backup.json file in the root of the sentry directory or in the scripts folder.
I'm confused.
This appears to be pretty strange behavior. If this is a new install on a fresh postgres db these integrity errors should not be occurring. @azaslavsky would you be able to take a look at this when you have a chance?
Self-Hosted Version
24.6.0
CPU Architecture
x86_64
Docker Version
27.0.3
Docker Compose Version
2.28.1
Steps to Reproduce
These are the steps I took:
./scripts/backup.sh global
./install.sh
from my Sentry directory (24.6.0)At this point I tried running the restore but encountered an error. I then tried running
docker compose up -d
and let it finish and tried running the restore but encountered another error. I then tried logging in to Sentry and then running the restore but it still fails.The documentation site states:
We strongly recommended that you restore your backup on the same version of Sentry on a fresh install (empty database but migrations are run)
So I assume this means that the restore should be run after
./install.sh
completes.But I'm unsure as to what migrations are or when they are run. Is this at the end of the install script? Is this after I call 'docker compose up -d`? Or is this after logging into the Sentry portal?
When exactly am I supposed to trigger the restore?
Expected Result
Expected a successful restore.
Actual Result
This is the restore log I see after a COMPLETELY FRESH INSTALL. This is after
./install.sh
completes and after runningdocker compose up -d
and waiting about 5 minutes.Event ID
No response