search

waliwali / ibm-db

Automatically exported from code.google.com/p/ibm-db
0 stars 0 forks source link

OpenStack database migrations fail with SQLAlchemy 0.9.6 and ibm-db-sa 0.3.1 #158

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
These are the packages used:

>>> sqlalchemy.__version__
'0.9.6'
>>> migrate.__version__
'0.9.1'
>>> ibm_db.__version__
'2.0.4.1'
>>> ibm_db_sa.__version__
'0.3.1'

Trying to run the openstack keystone database migration from master (there are 
other openstack projects that fail with similar errors on the same software 
levels) fails like this:

2014-07-09 02:34:14.488 | + /opt/stack/new/keystone/bin/keystone-manage db_sync
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 CRITICAL keystone [-] 
AttributeError: _all_cols
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone 
Traceback (most recent call last):
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/opt/stack/new/keystone/bin/keystone-manage", line 51, in <module>
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
cli.main(argv=sys.argv, config_files=config_files)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/opt/stack/new/keystone/keystone/cli.py", line 210, in main
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
CONF.command.cmd_class.main()
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/opt/stack/new/keystone/keystone/cli.py", line 72, in main
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
migration_helpers.sync_database_to_version(extension, version)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/opt/stack/new/keystone/keystone/common/sql/migration_helpers.py", line 160, 
in sync_database_to_version
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
init_version=init_version)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/migration.py", line 
79, in db_sync
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
return versioning_api.upgrade(engine, repository, version)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, 
in upgrade
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"<string>", line 2, in _migrate
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", 
line 160, in with_engine
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
return f(*a, **kw)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, 
in _migrate
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
schema.runchange(ver, change, changeset.step)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 93, 
in runchange
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
change.run(self.engine, step)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 
148, in run
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
script_func(engine)
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/opt/stack/new/keystone/keystone/common/sql/migrate_repo/versions/036_havana.py
", line 217, in upgrade
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
name='ixu_user_name_domain_id').create()
2014-07-09 02:36:03.614 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/constraint.py", line 
44, in create
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
self.__do_imports('constraintgenerator', *a, **kw)
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/constraint.py", line 
32, in __do_imports
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
run_single_visitor(engine, visitorcallable, self, *a, **kw)
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
 line 77, in run_single_visitor
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
fn(element, **kwargs)
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/ansisql.py", line 
294, in visit_migrate_unique_constraint
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
self._visit_constraint(*p, **k)
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/ibmdb2.py", 
line 268, in _visit_constraint
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone     for 
column in constraint.columns._all_cols:
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/_collections.py", line 
174, in __getattr__
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone     
raise AttributeError(key)
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone 
AttributeError: _all_cols
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone

Running with sqlalchemy 0.8.4 is successful.

This is the migration that fails:

http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/sql/migrat
e_repo/versions/036_havana.py

It's this UniqueConstraint creation that fails:

    migrate.UniqueConstraint(user.c.domain_id,
                             user.c.name,
                             name='ixu_user_name_domain_id').create()

Original issue reported on code.google.com by mattrie...@gmail.com on 11 Jul 2014 at 3:58

GoogleCodeExporter commented 9 years ago 
http://comments.gmane.org/gmane.comp.python.sqlalchemy.user/30027

Original comment by mattrie...@gmail.com on 14 Jul 2014 at 2:44

GoogleCodeExporter commented 9 years ago 
>>>
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone   File 
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/ibmdb2.py", 
line 268, in _visit_constraint
2014-07-09 02:36:03.615 | 2014-07-09 10:36:03.612 27124 TRACE keystone     for 
column in constraint.columns._all_cols:
<<<
above trace shows issue comes from sqlalchemy-migrate but similar issue also 
have with ibm_db_sa, I have make some changes to resolve this and committed the 
same in git branch.

I also have created a patch for sqlalchemy-migrate, you can find it in 
attachment.

Please give a try and let me know how it goes.

Original comment by rahul.pr...@in.ibm.com on 18 Jul 2014 at 9:36

Attachments:

GoogleCodeExporter commented 9 years ago

Original comment by rahul.pr...@in.ibm.com on 18 Jul 2014 at 9:50

GoogleCodeExporter commented 9 years ago 
Rahul, thanks, the sqlalchemy-migrate change is up for review here:

https://review.openstack.org/#/c/108129/

Original comment by mattrie...@gmail.com on 18 Jul 2014 at 6:45

GoogleCodeExporter commented 9 years ago 
The sqlalchemy-migrate patch didn't work because constraint.columns doesn't 
have a keys() method.

The updated patch is here: https://review.openstack.org/#/c/108129/4

UniqueConstraint inherits an __iter__ method which returns a tuple of the 
Column objects in the constraint so I just used that.

Original comment by mattrie...@gmail.com on 4 Aug 2014 at 2:38

GoogleCodeExporter commented 9 years ago 
I have tested the changes with SQLAlchemy-0.8.4 but with adding 
UniqueConstraint. You are facing the issue with adding FK constraint.

I will check with FK Constraint and will let you know if I face same issue.

Original comment by rahul.pr...@in.ibm.com on 6 Aug 2014 at 9:03

GoogleCodeExporter commented 9 years ago 
This is caused by adding FK constraint, It doesn't effect ibm_db_sa changes but 
I have modified the changes for code cleanup.

Original comment by rahul.pr...@in.ibm.com on 7 Aug 2014 at 8:39

GoogleCodeExporter commented 9 years ago 
@Rahul, OK, thanks for testing it out.

Original comment by mattrie...@gmail.com on 7 Aug 2014 at 2:02

GoogleCodeExporter commented 9 years ago 
Code changes released with ibm_db_sa-0.3.2

Original comment by rahul.pr...@in.ibm.com on 5 Nov 2014 at 6:33