ergo / ziggurat_foundations

Framework agnostic set of sqlalchemy classes that make building applications that require permissions an easy task.
BSD 3-Clause "New" or "Revised" License
71 stars 22 forks source link

Error on alembic migrations from docs #35

Closed arianmaykon closed 8 years ago

arianmaykon commented 8 years ago

Following https://ziggurat-foundations.readthedocs.org/en/latest/configuration.html#configure-ziggurat-with-pyramid-framework on the Wiki2 Pyramid Tutorial gives some errors on the initial migration step ($VENV/bin/alembic upgrade head).

I'm using Python 3.5.1 and Pyramid 1.6.1, here's the output:

(.venv) ➜  tutorial $VENV/bin/alembic upgrade head
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 2bb1ba973f0b, initial table layout
INFO  [alembic.runtime.migration] Running upgrade 2bb1ba973f0b -> 24ab8d11f014, add external identity tables
INFO  [alembic.runtime.migration] Running upgrade 24ab8d11f014 -> 5c84d7260c5, add id/parent id to resource structure
INFO  [alembic.runtime.migration] Running upgrade 5c84d7260c5 -> 46a9c4fb9560, make password hash field bigger
INFO  [alembic.runtime.migration] Running upgrade 46a9c4fb9560 -> 264049f80948, create ordering column
INFO  [alembic.runtime.migration] Running upgrade 264049f80948 -> 2d472fe79b95, bigger identity datatypes
Traceback (most recent call last):
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 217, in execute
    res = self._query(query)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 378, in _query
    rowcount = self._do_query(q)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 341, in _do_query
    db.query(q)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.DataError: (1171, 'All parts of a PRIMARY KEY must be NOT NULL; if you need NULL in a key, use UNIQUE instead')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/bin/alembic", line 9, in <module>
    load_entry_point('alembic', 'console_scripts', 'alembic')()
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/config.py", line 479, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/config.py", line 473, in main
    self.run_cmd(cfg, options)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/config.py", line 456, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/script/base.py", line 397, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/util/compat.py", line 68, in load_module_py
    module_id, path).load_module(module_id)
  File "<frozen importlib._bootstrap_external>", line 385, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 806, in load_module
  File "<frozen importlib._bootstrap_external>", line 665, in load_module
  File "<frozen importlib._bootstrap>", line 268, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 693, in _load
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 662, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Users/maykon/.python-eggs/ziggurat_foundations-0.6.2-py3.5.egg-tmp/ziggurat_foundations/migrations/env.py", line 85, in <module>
    run_migrations_online()
  File "/Users/maykon/.python-eggs/ziggurat_foundations-0.6.2-py3.5.egg-tmp/ziggurat_foundations/migrations/env.py", line 78, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/runtime/environment.py", line 797, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/runtime/migration.py", line 312, in run_migrations
    step.migration_fn(**kw)
  File "/Users/maykon/.python-eggs/ziggurat_foundations-0.6.2-py3.5.egg-tmp/ziggurat_foundations/migrations/versions/2d472fe79b95_bigger_identity_data.py", line 19, in upgrade
    type_=sa.String(255), existing_type=sa.String(50))
  File "<string>", line 8, in alter_column
  File "<string>", line 3, in alter_column
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/operations/ops.py", line 1414, in alter_column
    return operations.invoke(alt)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/operations/base.py", line 318, in invoke
    return fn(self, operation)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/operations/toimpl.py", line 53, in alter_column
    **operation.kw
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/ddl/mysql.py", line 67, in alter_column
    else existing_autoincrement
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/alembic-0.8.6-py3.5.egg/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 968, in _execute_ddl
    compiled
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 183, in reraise
    raise value.with_traceback(tb)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 217, in execute
    res = self._query(query)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 378, in _query
    rowcount = self._do_query(q)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/cursors.py", line 341, in _do_query
    db.query(q)
  File "/Users/maykon/Projects/pyramid_tutorial/.venv/lib/python3.5/site-packages/mysqlclient-1.3.7-py3.5-macosx-10.11-x86_64.egg/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.DataError: (_mysql_exceptions.DataError) (1171, 'All parts of a PRIMARY KEY must be NOT NULL; if you need NULL in a key, use UNIQUE instead') [SQL: 'ALTER TABLE external_identities MODIFY external_id VARCHAR(255) NULL']
ergo commented 8 years ago

I don't think wiki tutorial uses ziggurat? I will be looking into this issue as soon as possible, also @arianmaykon can you check if this happens on trunk version? I've been correcting some problems directly related to mysql.

ergo commented 8 years ago

@arianmaykon - I've released 0.6.3 on pypi - I ran all tests from this version on mysql and they all passed so I'm closing this.

Feel free to reopen if you still have problems on latest release.

arianmaykon commented 8 years ago

@ergo Ok, thanks a lot. I know the Wiki2 tutorial dont use ziggurat, but i was trying out with it. Thanks.

arianmaykon commented 8 years ago

@ergo sorry, i've upgraded zigurat but the error persists, i've made some fixes on the migrations (i know its not a good practice, but what can we do? I'm not very experienced with Python and its tools, maybe its my environment or my alembic version?

Here's the output of my pip list:

$VENV/bin/pip list
alembic (0.8.6)
anykeystore (0.2)
bcrypt (2.0.0)
beautifulsoup4 (4.4.1)
cffi (1.5.2)
Chameleon (2.24)
coverage (4.0.3)
defusedxml (0.4.1)
docutils (0.12)
Jinja2 (2.8)
Mako (1.0.4)
MarkupSafe (0.23)
mysqlclient (1.3.7)
oauthlib (1.0.3)
paginate (0.5.3)
paginate-sqlalchemy (0.2.0)
passlib (1.6.5)
PasteDeploy (1.5.2)
pip (8.1.1)
py (1.4.31)
pycparser (2.14)
Pygments (2.1.3)
pymlconf (0.3.21)
pyramid (1.6.1)
pyramid-basemodel (0.3.4)
pyramid-chameleon (0.3)
pyramid-debugtoolbar (2.4.2)
pyramid-fullauth (0.5.0)
pyramid-jinja2 (2.6.2)
pyramid-localize (0.1.0)
pyramid-mako (1.0.2)
pyramid-tm (0.12.1)
pytest (2.9.1)
pytest-cov (2.2.1)
python-editor (1.0)
python3-openid (3.0.10)
PyYAML (3.11)
repoze.lru (0.6)
requests (2.9.1)
requests-oauthlib (0.6.1)
setuptools (20.9.0)
six (1.10.0)
slugify (0.0.1)
SQLAlchemy (1.0.12)
transaction (1.4.4)
translationstring (1.3)
tzf.pyramid-yml (1.0.1)
velruse (1.1.1)
venusian (1.0)
waitress (0.9.0)
WebOb (1.6.0)
WebTest (2.0.21)
ziggurat-foundations (0.6.3)
zope.deprecation (4.1.2)
zope.interface (4.1.3)
zope.sqlalchemy (0.7.6)

I'll make a PR with the migrations fixes if you decide its a good idea to merge them. Let me know if you need anything.

ergo commented 8 years ago

What happens if you do:

DB_STRING="mysql+mysqlconnector://user:pass@127.0.0.1:3306/dbname" py.test ziggurat_foundations/tests.py

Are the tests passing for you?

ergo commented 8 years ago

I tried running the testsuite on Mysql 5.6.28 and all tests passed - including migrations.

arianmaykon commented 8 years ago

Sorry for this, but how do i run the tests? Did not find instructions on the repo files.

Here's what i have tryied:

cd ziggurat_foundations
pyenv shell 3.5.1
python -m venv .venv
.venv/bin/python setup.py develop
DB_STRING="mysql+mysqlconnector://root@127.0.0.1:3306/ziggurat" py.test ziggurat_foundations/tests.py

but it gives me:

zsh: command not found: py.test

its not in .venv/bin too.

I have tried that too:

./test.sh

but it returns:

find: -printf: unknown primary or operator
./test.sh: line 9: rstcheck: command not found

RST_CHECK: ./test.sh: line 9: rstcheck: command not found
FLAKE8: OK
ergo commented 8 years ago

You almost got it.

after you run the command: venv/bin/python setup.py develop run this: source venv/bin/activate this should active the virtual environment in your shell, then you need to install pip install http://cdn.mysql.com/Downloads/Connector-Python/mysql-connector-python-2.0.4.zip#md5=3df394d89300db95163f17c843ef49df to get the mysql driver from Oracle. DB_STRING="mysql+mysqlconnector://root@127.0.0.1:3306/ziggurat" py.test ziggurat_foundations/tests.py should work then.

ergo commented 8 years ago

Ok... I replicated the problem on 5.7.12, the tests stopped working.

ergo commented 8 years ago

@arianmaykon I've merged your fix, can you make a PR and update AUTHORS file with your name? If you do this I will release 0.6.4 to pypi. Thank you very much. I would still advise to use postgresql, they don't have that kind of weird changes across versions :-)

arianmaykon commented 8 years ago

@ergo i did run source venv/bin/activate but forgot to mention, sorry.

Ok, ill make the change, thanks a lot.

ergo commented 8 years ago

@arianmaykon 0.6.4 is now released to pypi. Thank you.