Backup import failure

[afaianswq]

Self-Hosted Version

23.12.1

CPU Architecture

x86_64

Docker Version

26.1.3

Docker Compose Version

2.27.0

Steps to Reproduce

1. cd /path/to/sentry-dist/sentry
2. Create a global backup: docker compose run --rm -T web export global > backup.json
3. Import backup: docker compose run --rm -T web import global --no-prompt /etc/sentry/backup.json

Expected Result

Backup imported without issues

Actual Result

Restore fails with the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
    rel_obj = self.field.get_cached_value(instance)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
KeyError: 'snuba_query'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/sentry/runner/__init__.py", line 195, in main
    func(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/sentry/runner/commands/backup.py", line 722, in import_global
    import_in_global_scope(
  File "/usr/local/lib/python3.10/site-packages/sentry/backup/imports.py", line 479, in import_in_global_scope
    return _import(
  File "/usr/local/lib/python3.10/site-packages/sentry/backup/imports.py", line 358, in _import
    _clear_model_tables_before_import()
  File "/usr/local/lib/python3.10/site-packages/sentry/backup/imports.py", line 60, in _clear_model_tables_before_import
    manager.all().delete()  # type: ignore
  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 156, in override
    return original_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 746, in delete
    deleted, _rows_count = collector.delete()
  File "/usr/local/lib/python3.10/site-packages/django/db/models/deletion.py", line 435, in delete
    signals.post_delete.send(
  File "/usr/local/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/usr/local/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/usr/local/lib/python3.10/site-packages/sentry/incidents/models.py", line 394, in clear_alert_rule_subscription_caches
    snuba_query=instance.snuba_query
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
    rel_obj = self.get_object(instance)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
sentry.snuba.models.SnubaQuery.DoesNotExist: SnubaQuery matching query does not exist.

Event ID

No response

[hubertdeng123]

This seems like some code that was written a while ago that does not directly influence the import process. Are you importing into a clean install of sentry? Is there information in your db?

cc @azaslavsky

[azaslavsky]

Is this error repeatable? I think it's a racy interaction of post-delete signals, and would work on the following try.

[afaianswq]

Yes, it's 100% reproducible.

[afaianswq]

This seems like some code that was written a while ago that does not directly influence the import process. Are you importing into a clean install of sentry? Is there information in your db?

cc @azaslavsky

I am importing into the same instance, so yes, there is information in the db.

[afaianswq]

I have just tested it with version 24.5.0 and the issue reproduced.

[hubertdeng123]

Are you also exporting from version 24.5.0? It's important to note that we only support imports from versions that are 3 versions from the version you're importing into. So, in your case, it would be 24.3.0

[afaianswq]

Yes, export/import is attempted on the same version always.

[azaslavsky]

You may need to drop this table manually. Is your instance currently working (ie, you failed to load the backup, but the instance still works as before)? Or is it completely bricked (which is what I would expect from this stack trace).

In any case, you'll need to use psql to drop the offending table manually - but please do a full backup first!

[afaianswq]

Which table do I need to drop? And why? I am not looking for a quick workaround. I want backup/restore feature to work reliably.

[azaslavsky]

I think I have a solution for this, will try to get it up by Monday.

[azaslavsky]

Update: it's a bit more involved than I had anticipated, so it may take a few days to get to the bottom of this and resolve. But I do believe this is fundamentally a bug on our end.

[shadowlmd]

Hello. Any news on this?

[azaslavsky]

Got a bit derailed by some other work, hoping to pick this up next week.

[shadowlmd]

Bump...

[shadowlmd]

Another bump. Can someone please address this quite critical issue?

[shadowlmd]

Bump

[shadowlmd]

Bump

[hubertdeng123]

Please keep in mind that this issue is on our radar, we just haven't gotten to it yet. Please avoid bumping this as it is not helpful to us for prioritization.

[shadowlmd]

Why is it still on your radar after 3 months? Restoring from a backup is a essential and critical part of the product, this bug is supposed to have critical priority.

[shadowlmd]

Any progress? This bug is holding us from upgrading from a more than one year old version.

[hubertdeng123]

No progress at the moment.

cc @azaslavsky