search

artefactory / one-click-mlflow

A tool to deploy a mostly serverless MLflow tracking server on a GCP project with one command
GNU Lesser General Public License v3.0
66 stars 21 forks source link

Error when running migrations #69

Closed griseau closed 3 years ago

griseau commented 3 years ago

Describe the bug Error when deploying mlflow server, 

pymysql.err.ProgrammingError: (1146, "Table 'mlflow.experiments' doesn't exist")
ValueError: Invalid IPv6 URL

To Reproduce Steps to reproduce the behavior: Just run make one-click-mlflow on an empty project

Full traceback

 Error: Error waiting to create FlexibleAppVersion: Error waiting for Creating FlexibleAppVersion: Error code 9, message: Flex operation projects/sandbox-thomas-323814/regions/europe-west1/operations/8cade860-1654-4007-91d4-a70f350bed1a error [FAILED_PRECONDITION]: An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2021-08-23T15:51:13.172Z56066.wm.0: 2021/08/23 15:52:57 INFO mlflow.store.db.utils: Updating database tables in preparation for MLflow 1.0 schema migrations 
│ INFO  [alembic.runtime.migration] Context impl MySQLImpl.
│ INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
│ INFO  [alembic.runtime.migration] Running upgrade  -> ff01da956556, ensure_unique_constraint_names
│ Traceback (most recent call last):
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
│     cursor, statement, parameters, context
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
│     cursor.execute(statement, parameters)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/cursors.py", line 163, in execute
│     result = self._query(query)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/cursors.py", line 321, in _query
│     conn.query(q)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 505, in query
│     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 724, in _read_query_result
│     result.read()
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 1069, in read
│     first_packet = self.connection._read_packet()
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 676, in _read_packet
│     packet.raise_for_error()
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/protocol.py", line 223, in raise_for_error
│     err.raise_mysql_exception(self._data)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
│     raise errorclass(errno, errval)
│ pymysql.err.ProgrammingError: (1146, "Table 'mlflow.experiments' doesn't exist")
│ 
│ The above exception was the direct cause of the following exception:
│ 
│ Traceback (most recent call last):
│   File "/usr/local/bin/mlflow", line 8, in <module>
│     sys.exit(cli())
│   File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1137, in __call__
│     return self.main(*args, **kwargs)
│   File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1062, in main
│     rv = self.invoke(ctx)
│   File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1668, in invoke
│     return _process_result(sub_ctx.command.invoke(sub_ctx))
│   File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1668, in invoke
│     return _process_result(sub_ctx.command.invoke(sub_ctx))
│   File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1404, in invoke
│     return ctx.invoke(self.callback, **ctx.params)
│   File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 763, in invoke
│     return __callback(*args, **kwargs)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/db.py", line 29, in upgrade
│     mlflow.store.db.utils._upgrade_db_initialized_before_mlflow_1(engine)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/store/db/utils.py", line 179, in _upgrade_db_initialized_before_mlflow_1
│     command.upgrade(config, "heads")
│   File "/usr/local/lib/python3.7/dist-packages/alembic/command.py", line 298, in upgrade
│     script.run_env()
│   File "/usr/local/lib/python3.7/dist-packages/alembic/script/base.py", line 489, in run_env
│     util.load_python_file(self.dir, "env.py")
│   File "/usr/local/lib/python3.7/dist-packages/alembic/util/pyfiles.py", line 98, in load_python_file
│     module = load_module_py(module_id, path)
│   File "/usr/local/lib/python3.7/dist-packages/alembic/util/compat.py", line 184, in load_module_py
│     spec.loader.exec_module(module)
│   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
│   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/temporary_db_migrations_for_pre_1_users/env.py", line 84, in <module>
│     run_migrations_online()
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/temporary_db_migrations_for_pre_1_users/env.py", line 78, in run_migrations_online
│     context.run_migrations()
│   File "<string>", line 8, in run_migrations
│   File "/usr/local/lib/python3.7/dist-packages/alembic/runtime/environment.py", line 846, in run_migrations
│     self.get_context().run_migrations(**kw)
│   File "/usr/local/lib/python3.7/dist-packages/alembic/runtime/migration.py", line 518, in run_migrations
│     step.migration_fn(**kw)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/temporary_db_migrations_for_pre_1_users/versions/ff01da956556_ensure_unique_constraint_names.py", line 180, in upgrade
│     condition=column("lifecycle_stage").in_(["active", "deleted"]),
│   File "/usr/lib/python3.7/contextlib.py", line 119, in __exit__
│     next(self.gen)
│   File "/usr/local/lib/python3.7/dist-packages/alembic/operations/base.py", line 354, in batch_alter_table
│     impl.flush()
│   File "/usr/local/lib/python3.7/dist-packages/alembic/operations/batch.py", line 83, in flush
│     fn(*arg, **kw)
│   File "/usr/local/lib/python3.7/dist-packages/alembic/ddl/impl.py", line 244, in add_constraint
│     self._exec(schema.AddConstraint(const))
│   File "/usr/local/lib/python3.7/dist-packages/alembic/ddl/impl.py", line 140, in _exec
│     return conn.execute(construct, *multiparams, **params)
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
│     return meth(self, multiparams, params)
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
│     return connection._execute_ddl(self, multiparams, params)
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1044, in _execute_ddl
│     compiled,
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
│     e, statement, parameters, cursor, context
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
│     util.raise_from_cause(sqlalchemy_exception, exc_info)
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
│     reraise(type(exception), exception, tb=exc_tb, cause=cause)
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
│     raise value.with_traceback(tb)
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
│     cursor, statement, parameters, context
│   File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
│     cursor.execute(statement, parameters)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/cursors.py", line 163, in execute
│     result = self._query(query)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/cursors.py", line 321, in _query
│     conn.query(q)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 505, in query
│     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 724, in _read_query_result
│     result.read()
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 1069, in read
│     first_packet = self.connection._read_packet()
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/connections.py", line 676, in _read_packet
│     packet.raise_for_error()
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/protocol.py", line 223, in raise_for_error
│     err.raise_mysql_exception(self._data)
│   File "/usr/local/lib/python3.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
│     raise errorclass(errno, errval)
│ sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'mlflow.experiments' doesn't exist")
│ [SQL: ALTER TABLE experiments ADD CONSTRAINT experiments_lifecycle_stage CHECK (lifecycle_stage IN ('active', 'deleted'))]
│ (Background on this error at: http://sqlalche.me/e/f405)
│ 2021/08/23 15:52:58 ERROR mlflow.cli: Error initializing backend store
│ 2021/08/23 15:52:58 ERROR mlflow.cli: Invalid IPv6 URL
│ Traceback (most recent call last):
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/cli.py", line 385, in server
│     initialize_backend_stores(backend_store_uri, default_artifact_root)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/server/handlers.py", line 146, in initialize_backend_stores
│     _get_tracking_store(backend_store_uri, default_artifact_root)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/server/handlers.py", line 131, in _get_tracking_store
│     _tracking_store = _tracking_store_registry.get_store(store_uri, artifact_root)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/_tracking_service/registry.py", line 37, in get_store
│     builder = self.get_store_builder(store_uri)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/registry.py", line 75, in get_store_builder
│     scheme = store_uri if store_uri == "databricks" else get_uri_scheme(store_uri)
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/utils/uri.py", line 157, in get_uri_scheme
│     scheme = urllib.parse.urlparse(uri_or_path).scheme
│   File "/usr/lib/python3.7/urllib/parse.py", line 368, in urlparse
│     splitresult = urlsplit(url, scheme, allow_fragments)
│   File "/usr/lib/python3.7/urllib/parse.py", line 459, in urlsplit
│     raise ValueError("Invalid IPv6 URL")
│ ValueError: Invalid IPv6 URL
│ 
│ 
│   with module.mlflow.module.server.google_app_engine_flexible_app_version.mlflow_app,
│   on modules/mlflow/server/main.tf line 112, in resource "google_app_engine_flexible_app_version" "mlflow_app":
│  112: resource "google_app_engine_flexible_app_version" "mlflow_app" {
│ 
╵
make: *** [apply-terraform] Error 1
AlexisVLRT commented 3 years ago

Well that's a new one... I'll check this out

pol-defont-reaulx commented 3 years ago

Hi @griseau I can't reproduce this error, is it possible for you to give me the following informations:

AlexisVLRT commented 3 years ago

I'm pretty sure Pol's first idea is right. And even if it's not the case for this specific bug, URL-illegal password can still be generated, so we need to fix this. @pol-defont-reaulx could you fix the password generation to exclude characters that can not be in an URL please ?

griseau commented 3 years ago

Hey @pol-defont-reaulx,

Thanks for having a look at it. Yes the password was y&R0x1Rq7@V]{0tW so you were right about this :)