Cannot get backup and restore to work

[drmrbrewer]

The restore section of the page relating to backups has the following statement:

This directory automatically gets mounted to /etc/sentry so you can run the following to restore your backup

But in my installation, this directory is not mounted to /etc/sentry... there is nothing at all mounted to /etc/sentry. So either there's a bug in the docs, or a bug in the installation.

[drmrbrewer]

If it means that the directory is mounted to /etc/sentry as part of the import process, and I just run that restore command as-is (with backup.json having been dumped into the sentry sub-folder within the main self-hosted repo folder ~/sentry/self-hosted), I just get the following errors:

~/sentry/self-hosted# docker compose run --rm -T web import /etc/sentry/backup.json
[+] Running 16/0
 ⠿ Container sentry-self-hosted-memcached-1                                 Running                                  0.0s
 ⠿ Container sentry-self-hosted-postgres-1                                  Running                                  0.0s
 ⠿ Container sentry-self-hosted-zookeeper-1                                 Running                                  0.0s
 ⠿ Container sentry-self-hosted-clickhouse-1                                Running                                  0.0s
 ⠿ Container sentry-self-hosted-kafka-1                                     Running                                  0.0s
 ⠿ Container sentry-self-hosted-symbolicator-1                              Running                                  0.0s
 ⠿ Container sentry-self-hosted-smtp-1                                      Running                                  0.0s
 ⠿ Container sentry-self-hosted-redis-1                                     Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-outcomes-consumer-1                   Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-sessions-consumer-1                   Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-consumer-1                            Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-replacer-1                            Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-api-1                                 Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-transactions-consumer-1               Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-subscription-consumer-events-1        Running                                  0.0s
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
11:20:16 [INFO] sentry.plugins.github: apps-not-configured
/usr/local/lib/python3.8/site-packages/sentry/runner/initializer.py:551: DeprecatedSettingWarning: The SENTRY_URL_PREFIX setting is deprecated. Please use SENTRY_OPTIONS['system.url-prefix'] instead.
  warnings.warn(DeprecatedSettingWarning(old, "SENTRY_OPTIONS['%s']" % new))
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key_extra_len is 0:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer
    objects = json.loads(stream_or_string)
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 188, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import_
    for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 73, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError

[AbhiPrasad]

Hey thanks for writing in. I'm going to transfer this issue to the self-hosted repo so the folks who work on that can help out!

[drmrbrewer]

Thanks, @AbhiPrasad.

@chadwhitacre I may be doing something wrong, but I'd expect that (as a test) I should be able to do a backup followed immediately by a restore of that backup, based on the steps as described on the Self-Hosted Backup & Restore page (using docker compose rather than docker-compose):

Backup:

docker compose run --rm -T -e SENTRY_LOG_LEVEL=CRITICAL web export > sentry/backup.json

Restore:

docker compose run --rm -T web import /etc/sentry/backup.json

But I just get the error messages as shown previously when I run the restore command.

[chadwhitacre]

But in my installation, this directory is not mounted to /etc/sentry... there is nothing at all mounted to /etc/sentry

Quick skim (and recollection based on experience) suggests this is unexpected, afaict we mount /etc/sentry for all containers that use the x-sentry-defaults config:

https://github.com/getsentry/self-hosted/blob/25381dab2e56115a17798850b139862cc3c09ea6/docker-compose.yml#L69

[chadwhitacre]

I should be able to do a backup followed immediately by a restore

The export/import functionality is somewhat undermaintained, so it wouldn't surprise me if it's broken. I'm not sure what testing we have for it over in sentry, a high-level integration test for the basic functionality would be good to have if we don't already.

[drmrbrewer]

this is unexpected, afaict we mount /etc/sentry for all containers that use the x-sentry-defaults config:

# ls /etc/sentry
ls: cannot access '/etc/sentry': No such file or directory

The sentry installation is otherwise working fine.

The export/import functionality is somewhat undermaintained, so it wouldn't surprise me if it's broken.

Would be great to get this working properly because in IMHO backup and restore is a key feature.

[chadwhitacre]

backup and restore is a key feature

Since it's only metadata and not actual event data, it's less critical than it would be if it were a full backup solution. For that we bump to Docker.

[drmrbrewer]

Since it's only metadata and not actual event data

For me, metadata is more important (to back up) than actual event data, because event data will just repopulate over time but metadata won't.

[drmrbrewer]

@chadwhitacre did you confirm this as a bug, or is it just something I'm misunderstanding about the backup and restore process?

[chadwhitacre]

Have not had time to confirm, no. You made it past the original /etc/sentry mount issue, yes? Now we're on JSONDecodeError during restore? Did you inspect the contents of the json file to see if there's any clues as to what is causing the decode error?

[drmrbrewer]

You made it past the original /etc/sentry mount issue, yes?

No, still nothing is mounted there:

# ls /etc/sentry
ls: cannot access '/etc/sentry': No such file or directory

But everything else is working fine without that.

[drmrbrewer]

@chadwhitacre still not figured how to get this working :-(

Referring to your comment, which containers actually use the x-sentry-defaults config?

[chadwhitacre]

still not figured how to get this working :-(

D'oh! :-/ I need to wrap my head around this subsystem ... won't be able to fit that in this week but maybe next week? 🤞

which containers actually use the x-sentry-defaults config?

Grep for sentry_defaults in docker-compose.yml.

https://github.com/getsentry/self-hosted/blob/8c5c1adf956ed589d53967447d567cd65dbadf84/docker-compose.yml#L17

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[chadwhitacre]

Still interested in this @drmrbrewer?

[drmrbrewer]

@chadwhitacre yes, still very interested!

[chadwhitacre]

Okay, in that case I'll try to convince the bot to leave this ticket alone. :)

[chadwhitacre]

From @miraslav @onmomo in https://github.com/getsentry/self-hosted/issues/1428:

---

Ok, how I can resolve that error?

File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 73, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError

---

The sentry export command generated an invalid json for me

root@sentry-web-55cd99ffd9-628vf:/# sentry export -> export.json

resulted in the following

06:15:02 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
06:15:10 [INFO] sentry.plugins.github: apps-not-configured
[
{
  "model": "sites.site",
  "pk": 1,
  "fields": {
    "domain": "example.com",
    "name": "example.com"
  }
},
....

Removing the log messages in the beginning of the file, fixed the json.decoder.JSONDecodeError: Extra data: line 1 column 2 (char 1) error. Which corresponds to what is documented here.

• https://develop.sentry.dev/self-hosted/backup/#backup

[andreasschmitz]

The messages at the beginning of the file are not part of the actual export command, but something that is printed when the docker container is started. To be honest, I haven't figured out that part yet.

However, that allows us to work around the problem by mounting a folder into the docker container and exporting the backup file directly into it.

For example, the following mounts the current folder into the docker container at path /backup and then exports into the mounted folder.

docker-compose run --rm -T -e SENTRY_LOG_LEVEL=CRITICAL -v $(pwd):/backup web bash -c "sentry export -q /backup/backup.json"

In this case the SENTRY_LOG_LEVEL=CRITICAL part isn't even needed anymore, at least in my tests.

Hope that helps mitigate the problem :). The import should work as documented.

In my view this is a less error-prone way to achieve the same thing. Maybe worth updating the documentation for?

[chadwhitacre]

Closing in favor of https://github.com/getsentry/sentry/issues/36868 since this functionality lives in that repo.