getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
39.06k stars 4.19k forks source link

Migration error with MySQL store #1354

Closed josephmc5 closed 9 years ago

josephmc5 commented 9 years ago

Fresh database against master at b4ce969a8432dae4b42dae16dd90fc1dbcabe84a.

I know you guys don't use MySQL yourself but I thought I would mention it to see if you had any ideas. I'm looking into it myself.

 > sentry:0129_auto__chg_field_release_id__chg_field_pendingteammember_id__chg_field_
 > sentry:0130_auto__del_field_project_owner
 > sentry:0131_auto__add_organizationmember__add_unique_organizationmember_organizati
FATAL ERROR - The following SQL query failed: ALTER TABLE `sentry_organizationmember` ADD CONSTRAINT `organization_id_refs_id_42dc8e8f` FOREIGN KEY (`organization_id`) REFERENCES `sentry_organization` (`id`);
The error was: (1005, "Can't create table 'sentry.#sql-7684_768e' (errno: 150)")
 ! Error found during real run of migration! Aborting.

 ! Since you have a database that does not support running
 ! schema-altering statements in transactions, we have had
 ! to leave it in an interim state between migrations.

! You *might* be able to recover with:   - no dry run output for delete_unique_column() due to dynamic DDL, sorry
   = DROP TABLE `sentry_organizationmember` CASCADE; []
   = DROP TABLE `sentry_organization` CASCADE; []
   - no dry run output for delete_foreign_key() due to dynamic DDL, sorry
   = ALTER TABLE `sentry_team` DROP COLUMN `organization_id` CASCADE; []

 ! The South developers regret this has happened, and would
 ! like to gently persuade you to consider a slightly
 ! easier-to-deal-with DBMS (one that supports DDL transactions)
 ! NOTE: The error which caused the migration to fail is further up.
Error in migration: sentry:0131_auto__add_organizationmember__add_unique_organizationmember_organizati
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 9, in <module>
    load_entry_point('sentry==7.0.0-DEV', 'console_scripts', 'sentry')()
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/sentry/utils/runner.py", line 381, in main
    initializer=initialize_app,
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/logan/runner.py", line 169, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/sentry/management/commands/upgrade.py", line 29, in handle
    call_command('syncdb', migrate=True, interactive=(not options['noinput']))
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/south/management/commands/syncdb.py", line 103, in handle_noargs
    management.call_command('migrate', **options)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/raven/contrib/django/management/__init__.py", line 37, in new_execute
    return original_func(self, *args, **kwargs)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
    ignore_ghosts = ignore_ghosts,
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/south/migration/migrators.py", line 256, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/south/migration/migrators.py", line 337, in migrate_many
    interactive=self.interactive)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/south/db/generic.py", line 1030, in send_pending_create_signals
    interactive=interactive)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/south/db/generic.py", line 1076, in really_send_create_signal
    db=self.db_alias,
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 185, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/sentry/receivers/core.py", line 34, in create_default_projects
    platform='django',
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/sentry/receivers/core.py", line 63, in create_default_project
    'owner': user,
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/query.py", line 376, in get_or_create
    return self.get(**lookup), False
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/query.py", line 304, in get
    num = len(clone)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/query.py", line 857, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/sentry/env_27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
django.db.utils.OperationalError: (1054, "Unknown column 'sentry_organization.slug' in 'field list'")
josephmc5 commented 9 years ago

Updated the error to include more.

josephmc5 commented 9 years ago

I've isolated the cause of the error. organization.id is a bigint but the FK in organizationmember gets created as an int. When the alter statement to create the constraint gets called MySQL fails because the two columns are a different type.

sentry_organization

CREATE TABLE `sentry_organization` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  `owner_id` int(11) NOT NULL,
  `status` int(10) unsigned NOT NULL,
  `date_added` datetime NOT NULL,
  PRIMARY KEY (`id`)
)

sentry_organizationmember

CREATE TABLE `sentry_organizationmember` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `organization_id` int(11) NOT NULL,
  `user_id` int(11) NOT NULL,
  `type` int(10) unsigned NOT NULL,
  `date_added` datetime NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `sentry_organizationmember_organization_id_404770fc5e3a794_uniq` (`organization_id`,`user_id`)
)

Running migrations for sentry:

 - Migrating forwards to 0148_auto__add_helppage.
 > sentry:0131_auto__add_organizationmember__add_unique_organizationmember_organizati
   = SET FOREIGN_KEY_CHECKS=0; []
   = CREATE TABLE `sentry_organizationmember` (`id` bigint AUTO_INCREMENT NOT NULL PRIMARY KEY, `organization_id` integer NOT NULL, `user_id` integer NOT NULL, `type` integer UNSIGNED NOT NULL, `date_added` datetime NOT NULL) []
   = ALTER TABLE `sentry_organizationmember` ADD CONSTRAINT `sentry_organizationmember_organization_id_404770fc5e3a794_uniq` UNIQUE (`organization_id`, `user_id`) []
   = CREATE TABLE `sentry_organization` (`id` bigint AUTO_INCREMENT NOT NULL PRIMARY KEY, `name` varchar(64) NOT NULL, `owner_id` integer NOT NULL, `status` integer UNSIGNED NOT NULL, `date_added` datetime NOT NULL) []
   = ALTER TABLE `sentry_team` ADD COLUMN `organization_id` integer NULL; []
   - no dry run output for alter_column() due to dynamic DDL, sorry
   = ALTER TABLE `sentry_organizationmember` ADD CONSTRAINT `organization_id_refs_id_42dc8e8f` FOREIGN KEY (`organization_id`) REFERENCES `sentry_organization` (`id`); []
   = CREATE INDEX `sentry_organizationmember_de772da3` ON `sentry_organizationmember` (`organization_id`); []
   = ALTER TABLE `sentry_organizationmember` ADD CONSTRAINT `user_id_refs_id_be455e60` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`); []
   = CREATE INDEX `sentry_organizationmember_6340c63c` ON `sentry_organizationmember` (`user_id`); []
   = ALTER TABLE `sentry_organization` ADD CONSTRAINT `owner_id_refs_id_92f97975` FOREIGN KEY (`owner_id`) REFERENCES `auth_user` (`id`); []
   = CREATE INDEX `sentry_organization_cb902d83` ON `sentry_organization` (`owner_id`); []
   = ALTER TABLE `sentry_team` ADD CONSTRAINT `organization_id_refs_id_61038a42` FOREIGN KEY (`organization_id`) REFERENCES `sentry_organization` (`id`); []
   = CREATE INDEX `sentry_team_de772da3` ON `sentry_team` (`organization_id`); []

and replicated problem

mysql> ALTER TABLE `sentry_organizationmember` ADD CONSTRAINT `organization_id_refs_id_42dc8e8f` FOREIGN KEY (`organization_id`) REFERENCES `sentry_organization` (`id`);
ERROR 1005 (HY000): Can't create table 'sentry.#sql-7684_7e2b' (errno: 150)

I've been trying so hard to make it run but I just can't with my limited knowledge of South. ANY help would be hugely appreciated.

dcramer commented 9 years ago

@josephmc5 thanks for debugging this!

I'm going to dig into it a bit now they identified the root cause. Seems it might be non trivial but we can for sure fix it.

https://groups.google.com/forum/#!topic/south-users/X1_8ezSaLAI

dcramer commented 9 years ago

My ego convinced me I could solve this in 5m, and it turns out its become quite the pain. If this is a new install, I'd highly suggest Postgres, but if you can't, you can:

$ sentry sql sentry

We do a shitty job at indexes, but this will dump the raw SQL (which you may want to tweak).

Once thats done you can:

$ sentry migrate sentry --fake

That said, I'll probably have a solution for this today, as now I'm convinced its solvable.

dcramer commented 9 years ago

@josephmc5 feel like trying out f34cf31 ?

josephmc5 commented 9 years ago

Thank you for looking into it and providing a solution! It's not a new install but I'm installing from scratch on my dev box to do testing. Ok So the migrations didn't finish with that changeset! My only problem now is that the static assets aren't getting delivered right and I'm getting a page not found from sentry. That seems unrelated to these issues though.

dcramer commented 9 years ago

Which static assets are 404'ing? Also keep in mind that if you're installing from master we now require node.js be installed in order to build assets.

josephmc5 commented 9 years ago

It appears that all of them are. I do have node.js installed. Is there a way I can verify that the assets were generated or try to regenerate them?

I should note that I'm installing with pip install "repo tarball"

josephmc5 commented 9 years ago

@dcramer ok I think it's just because python setup.py install doesn't build static. I got it working now by doing a

python setup.py sdist --formats=gztar

then installing that gzip

pip install --force-reinstall dist/sentry-7.1.0.dev0.tar.gz

dcramer commented 9 years ago

@josephmc5 was this an install from source? You're right that it doesn't. I forgot we killed it for reasons.

I think in general what we are going to need to do is just make it smarter.

Here's the rules I'm thinking:

josephmc5 commented 9 years ago

@dcramer yeah I have my own fork to add ldap support. It basically just adds a hook on login to add all users to all projects except sentry.

I agree it should at least give a warning. Perhaps some documentation too. Although the the develop setup.py option did work.