search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.79k stars 1.76k forks source link

Restore feature not working - django.core.serializers.base.DeserializationError #1865

Closed mesenger closed 1 year ago

mesenger commented 1 year ago

Self-Hosted Version

22.12.0.dev0+99926859321bd1f82d1ac4b07bc8601e3d78b9c9

CPU Architecture

x86_64

Docker Version

20.10.21

Docker Compose Version

1.29.2

Steps to Reproduce

  1. Take backup of existing Sentry installation - OK
  2. Remove Sentry, purge volumes, install same version ./install.sh --skip-user-prompt - OK
  3. try to restore previous backup: docker-compose run --rm -T web import /etc/sentry/backup.json - Error: docker-compose run --rm -T web import /etc/sentry/backup.json Starting sentry-self-hosted_symbolicator_1 ... done Starting sentry-self-hosted_postgres_1 ... done Starting sentry-self-hosted_zookeeper_1 ... done Starting sentry-self-hosted_clickhouse_1 ... done Starting sentry-self-hosted_memcached_1 ... done Starting sentry-self-hosted_smtp_1 ... done Starting sentry-self-hosted_redis_1 ... done Starting sentry-self-hosted_kafka_1 ... done Starting sentry-self-hosted_snuba-subscription-consumer-transactions_1 ... done Starting sentry-self-hosted_snuba-replacer_1 ... done Starting sentry-self-hosted_snuba-sessions-consumer_1 ... done Starting sentry-self-hosted_snuba-consumer_1 ... done Starting sentry-self-hosted_snuba-api_1 ... done Starting sentry-self-hosted_snuba-transactions-consumer_1 ... done Starting sentry-self-hosted_snuba-outcomes-consumer_1 ... done Starting sentry-self-hosted_snuba-subscription-consumer-events_1 ... done Creating sentry-self-hosted_web_run ... done Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. 15:06:27 [INFO] sentry.plugins.github: apps-not-configured Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer objects = json.loads(stream_or_string) File "/usr/local/lib/python3.8/json/init.py", line 357, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/bin/sentry", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 192, in main func(kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner return ctx.invoke(f, *args, *kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import_ for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True): File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 73, in Deserializer raise DeserializationError() from exc django.core.serializers.base.DeserializationError ERROR: 1

Unable to restore

Expected Result

Restore would finish OK and project data would be restored correctly

Actual Result

backup is not restored, producing error:

15:06:27 [INFO] sentry.plugins.github: apps-not-configured Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer objects = json.loads(stream_or_string) File "/usr/local/lib/python3.8/json/init.py", line 357, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/bin/sentry", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 192, in main func(kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner return ctx.invoke(f, *args, *kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import_ for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True): File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 73, in Deserializer raise DeserializationError() from exc django.core.serializers.base.DeserializationError ERROR: 1

Event ID

No response

ethanhs commented 1 year ago

Did you check the output of the generated JSON file? It seems that something expected is missing.

mesenger commented 1 year ago

I am not an expert on json syntax, it looks good enough to me. jq reports some errors. Here are some snippets : 

root@sentry1:~/sentry# jq sentry/backup.json
jq: error: sentry/0 is not defined at <top-level>, line 1:
sentry/backup.json
jq: error: backup/0 is not defined at <top-level>, line 1:
sentry/backup.json       
jq: 2 compile errors

root@sentry1:~/sentry# head sentry/backup.json 
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
[
{
  "model": "sites.site",
  "pk": 1,
  "fields": {
    "domain": "example.com",

root@sentry1:~/sentry# tail sentry/backup.json
{
  "model": "sentry.projectoption",
  "pk": 7,
  "fields": {
    "project": 2,
    "key": "sentry:option-epoch",
    "value": 8
  }
}
]
mesenger commented 1 year ago

I removed the top part from backup.json, starting now with this: [ { "model": "sites.site", "pk": 1, "fields": { "domain": "example.com",

now I get more specific error:

`13:44:04 [INFO] sentry.plugins.github: apps-not-configured Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 143, in Deserializer data[field.name] = field.to_python(field_value) File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/picklefield.py", line 54, in to_python return json.loads(value) File "/usr/local/lib/python3.8/site-packages/sentry/utils/json.py", line 120, in loads return _default_decoder.decode(value) File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 392, in raw_decode raise TypeError("Input string must be text, not bytes") TypeError: Input string must be text, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/sentry", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 192, in main func(kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner return ctx.invoke(f, *args, *kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import_ for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True): File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 69, in Deserializer yield from PythonDeserializer(objects, **options) File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 145, in Deserializer raise base.DeserializationError.WithData(e, d['model'], d.get('pk'), field_value) django.core.serializers.base.DeserializationError: Input string must be text, not bytes: (sentry.option:pk=1) field_value was '1670942824.3678975' ERROR: 1 `

I can see this entry in backup.json, but not sure what could be wrong with it: { "model": "sentry.option", "pk": 1, "fields": { "key": "sentry:last_worker_ping", "value": 1670942824.3678975, "last_updated": "2022-12-13T14:47:04.367Z" } },

ethanhs commented 1 year ago

Thanks for digging into that! In that case, I believe this might be the same as https://github.com/getsentry/sentry/issues/36868 I will close this in favor of that issue. Unfortunately right now we don't have the bandwidth to fix the backup, but we likely will tackle it Q1.