search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.81k stars 1.76k forks source link

column sentry_notificationsetting.target_id_id does not exist #2281

Closed Trepliev closed 1 year ago

Trepliev commented 1 year ago

Self-Hosted Version

23.7.0

CPU Architecture

x86_64

Docker Version

20.10.23

Docker Compose Version

2.19.1

Steps to Reproduce

I tried to upgrade sentry from 23.4.0 to 23.7.0. The process crashed during db migrations with error. The sentry is unusable now.

Oops! Something went wrong It looks like you've hit an issue in our server API. Don't worry though! We use Sentry to monitor Sentry and it's likely we're already looking into this!

If you're daring, you may want to try the following:

Give it a few seconds and reload the page. If all else fails, contact us with more details.

Expected Result

Successful upgrade.

Actual Result

Running migrations for default
Operations to perform:
  Apply all migrations: auth, contenttypes, nodestore, replays, sentry, sessions, sites, social_auth
Running migrations:
Notification Settings: 100% |#                                 | ETA:  --:--:--Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UndefinedColumn: column sentry_notificationsetting.target_id_id does not exist
LINE 1: ...team_id", "sentry_notificationsetting"."user_id", "sentry_no...
                                                             ^
HINT:  Perhaps you meant to reference the column "sentry_notificationsetting.target_id".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UndefinedColumn: UndefinedColumn('column sentry_notificationsetting.target_id_id does not exist\nLINE 1: ...team_id", "sentry_notificationsetting"."user_id", "sentry_no...\n                                                             ^\nHINT:  Perhaps you meant to reference the column "sentry_notificationsetting.target_id".\n')
SQL: SELECT "sentry_notificationsetting"."id", "sentry_notificationsetting"."scope_type", "sentry_notificationsetting"."scope_identifier", "sentry_notificationsetting"."provider", "sentry_notificationsetting"."type", "sentry_notificationsetting"."value", "sentry_notificationsetting"."team_id", "sentry_notificationsetting"."user_id", "sentry_notificationsetting"."target_id_id" FROM "sentry_notificationsetting" WHERE ("sentry_notificationsetting"."team_id" IS NULL AND "sentry_notificationsetting"."user_id" IS NULL) ORDER BY "sentry_notificationsetting"."id" ASC LIMIT 1000

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 117, in upgrade
    _upgrade(
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 52, in _upgrade
    dj_call_command(
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/monkey/executor.py", line 79, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/migrations/0439_backfill_notificationsetting.py", line 18, in backfill_notificationsetting
    for setting in RangeQuerySetWrapperWithProgressBar(queryset):
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/query.py", line 233, in __iter__
    for idx, item in enumerate(self.iterator):
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/query.py", line 147, in __iter__
    results = list(results[0 : self.step])
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 616, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.ProgrammingError: UndefinedColumn('column sentry_notificationsetting.target_id_id does not exist\nLINE 1: ...team_id", "sentry_notificationsetting"."user_id", "sentry_no...\n                                                             ^\nHINT:  Perhaps you meant to reference the column "sentry_notificationsetting.target_id".\n')
SQL: SELECT "sentry_notificationsetting"."id", "sentry_notificationsetting"."scope_type", "sentry_notificationsetting"."scope_identifier", "sentry_notificationsetting"."provider", "sentry_notificationsetting"."type", "sentry_notificationsetting"."value", "sentry_notificationsetting"."team_id", "sentry_notificationsetting"."user_id", "sentry_notificationsetting"."target_id_id" FROM "sentry_notificationsetting" WHERE ("sentry_notificationsetting"."team_id" IS NULL AND "sentry_notificationsetting"."user_id" IS NULL) ORDER BY "sentry_notificationsetting"."id" ASC LIMIT 1000
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12

Cleaning up...

Event ID

No response

azaslavsky commented 1 year ago

@vbro Is there any chance this is related to migration 0434? That seems to be the last time we touched this model - maybe in all the renaming of target -> target_id, something got mixed up?

max312 commented 1 year ago

i encountered the same error. the log doesn't even provide which migration step fails, because it errors before logging the actual migration.

max312 commented 1 year ago

I created "sentry_notificationsettings.target_id_id" manually in postgres to skip through the error. The migrations continued with

Notification Settings: 100% |#                                 | ETA:  --:--:--
 OK
  Applying sentry.0440_add_control_option... OK`

but this lead to another error in migration 0503

`Applying sentry.0503_alter_notification_actor_nullable...Traceback (most recent call last):

  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/schema.py", line 52, in inner
    func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 608, in alter_field
    self._alter_field(model, old_field, new_field, old_type, new_type,
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/postgresql/schema.py", line 196, in _alter_field
    super()._alter_field(
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 705, in _alter_field
    self.execute(self._rename_field_sql(model._meta.db_table, old_field, new_field, new_type))
  File "/usr/local/lib/python3.8/site-packages/django_zero_downtime_migrations/backends/postgres/schema.py", line 415, in _rename_field_sql
    raise UnsafeOperationException(Unsafe.ALTER_TABLE_RENAME_COLUMN)
django_zero_downtime_migrations.backends.postgres.schema.UnsafeOperationException: ALTER TABLE RENAME COLUMN is unsafe operation
See details for save alternative https://github.com/tbicr/django-pg-zero-downtime-migrations#changes-for-working-logic

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 117, in upgrade
    _upgrade(
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 52, in _upgrade
    dj_call_command(
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/monkey/executor.py", line 79, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 244, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/schema.py", line 60, in inner
    raise UnsafeOperationException(exc_str.format(*formatted_args))
django_zero_downtime_migrations.backends.postgres.schema.UnsafeOperationException: Renaming column NotificationSetting.target_id to target_id is unsafe.`
mmerickel commented 1 year ago

Just ran into this error too after waiting several hours for the database to upgrade. :-( Forced to reinstall from backup with some data loss....

<snip>
  Applying sentry.0435_add_alert_rule_source... OK
  Applying sentry.0436_break_fks_on_external_actor_and_issue... OK
  Applying sentry.0437_remove_fk_notifications_target... OK
  Applying sentry.0438_break_inviter_fk_organizationmember... OK
Notification Settings: 100% |#                                 | ETA:  --:--:--Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UndefinedColumn: column sentry_notificationsetting.target_id_id does not exist
LINE 1: ...team_id", "sentry_notificationsetting"."user_id", "sentry_no...
                                                             ^
HINT:  Perhaps you meant to reference the column "sentry_notificationsetting.target_id".
juslintek commented 1 year ago

To fix this do: git checkout tags/23.6.2 -b 23.6.2 and then run ./install.sh :-)

Worked for me. Had exactly the same problem. :-) All data is there. I guess bug was introduced in 23.7.0 :-)

mmerickel commented 1 year ago

Can anyone from Sentry attest to this rollback being supported at least in theory?

  1. I'm very afraid of leaving the database in an inconsistent state where it might fail to upgrade in the future after a fix is rolled out.
  2. The 23.6.2 codebase may not work with the schema in the state that it is in - partially upgraded to the 23.7.0 codebase and something will be subtly broken.
markstory commented 1 year ago

@mmerickel Going from 23.4.0 to 23.6.2 should be fine. If your migrations are failing at 0438, you won't have gotten to a point in the migration history where you couldn't roll back to 23.6.

I'll take a look into why the target_id_id field alias is being used. That column name wasn't one we ever had.

juslintek commented 1 year ago

Regarding all above, for me it was, that I'm always on nightly and this time it went wrong. Always on the edge. :-) Did install.sh couple of times, upgraded docker to latest version. OS to latest version. Nothing helped. Then decided to go rejection path, checkout to versions till it works. My version before upgrade was 23.3.1. Also during upgrade my HDD on EC2 Volume storage got out of space. So I just improved its size from 160GB to 320GB and increase IOPS to 4000 and increased transfer rate to 1000MiB/s. After that waited till it will complete optimisation, extended volume on Ubuntu OS. And tried again, but it failed. So checkout to 23.6.2 ran install.sh and it finished upgrade. Also was looking at the changes between 23.6.2 and 23.7.0 on sentry itself project, where migrations are, but was not able to spot where that _id is appended. There are some files which contain more than 7k lines... So either need to clone whole project to my IDE to lookup with files and also setup deb environment and run migrations with debugger, or just leave it be and wait for the fix from people who already have everything setup and know project by heart. :-D

glatzor commented 1 year ago

Upgrading from 23.4.0 to 2.5.0 and then to 2.6.0 and finally to 2.7.0 fixed the issue for me.

tsc-awardle commented 1 year ago

I just ran into this issue upgrading from 22.3.0 to 23.7.0.

I tried the suggestions of running ./install.sh from the 23.6.2 and 23.5.0 branches and received the same error, Not sure how @juslintek was able to get it working without restoring from backup.

Do I need to pin the docker image to an earlier build?

EDIT: I realized my .env.custom file was defaulting to nightly image still. I pinned the images to 23.5.0 and was able to upgrade after that.

azaslavsky commented 1 year ago

We've got a fix in the pipeline. We will try to roll the referenced change into the sentry repo today, and then push 23.7.1 out on Monday.

nkeilar commented 1 year ago

Pleased I took a snapshot before upgrading, from 23.6.2 to 23.7.0 as hit this error too. Left DB in degraded state. Recommend not upgrading until this is resolved.

macteo commented 1 year ago

We've got a fix in the pipeline. We will try to roll the referenced change into the sentry repo today, and then push 23.7.1 out on Monday.

We're stuck since yesterday morning (UTC) and due the aforementioned planned release we decided to wait. Any news about the 23.7.1 release date? Thanks

azaslavsky commented 1 year ago

We're aiming to cut the release today.

azaslavsky commented 1 year ago

The release is currently being rolled, tracking bug is here. It should be up in 3-4 hours.

azaslavsky commented 1 year ago

23.7.1 has landed. Marking this closed, though if anyone still has trouble moving from an uncorrupted database on 23.6.x or below to 23.7.1, please update or file a new bug.

macteo commented 1 year ago

Sorry to bother you, but clearly I intended that the fix would be able to salvage a botched 23.7.0 update. Right now I'm stuck with another migration not running correctly: 0468_pickle_to_json_sentry_rawevent.py.

Thanks for your help

Running migrations for default
Operations to perform:
  Apply all migrations: auth, contenttypes, nodestore, replays, sentry, sessions, sites, social_auth
Running migrations:
Raw Events: N/A% |                                             | ETA:  --:--:--Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 117, in upgrade
    _upgrade(
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 52, in _upgrade
    dj_call_command(
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/monkey/executor.py", line 79, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/sentry/migrations/0468_pickle_to_json_sentry_rawevent.py", line 14, in _backfill
    obj.save(update_fields=["data"])
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 858, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1162, in execute_sql
    sql, params = self.as_sql()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1525, in as_sql
    val = field.get_db_prep_save(val, connection=self.connection)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 842, in get_db_prep_save
    return self.get_db_prep_value(value, connection=connection, prepared=False)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 837, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/node.py", line 242, in get_prep_value
    value.save()
  File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/node.py", line 156, in save
    nodestore.set_subkeys(self.id, subkeys)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 135, in <lambda>
    context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py", line 199, in inner
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/nodestore/base.py", line 245, in set_subkeys
    bytes_data = self._encode(data)
  File "/usr/local/lib/python3.8/site-packages/sentry/nodestore/base.py", line 204, in _encode
    lines = [json_dumps(data.pop(None)).encode("utf8")]
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/simplejson/encoder.py", line 272, in default
    raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type NodeData is not JSON serializable
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12