SyntaxError in migration 2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py

grutts commented 6 months ago

Bug description

We are in the process of upgrading from 1.5.1 towards 3.1.0.

When attempting to apply the migration 2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py with AthenaDialect the migration throws an error SyntaxError: EOL while scanning string literal

How to reproduce the bug

Attempt to migrate 2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py from a database running version 1.5.1 Apply the migration from a container running version 3.1.0

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

16

Browser

Chrome

Additional context

1709160927311,INFO [alembic.runtime.migration] Context impl PostgresqlImpl. 1709160927311,INFO [alembic.runtime.migration] Will assume transactional DDL. 1709160928059,"INFO [alembic.runtime.migration] Running upgrade 58df9d617f14 -> 2ed890b36b94, rm_time_range_endpoints_from_qc" 1709160928383,"INFO [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> b0d0249074e4, deprecate time_range_endpoints v2" 1709160928600,"INFO [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> 8b841273bec3, sql_lab_models_database_constraint_updates" 1709160928664,"INFO [alembic.runtime.migration] Running upgrade 8b841273bec3, b0d0249074e4 -> 9d8a8d575284, merge point" 1709160928669,"INFO [alembic.runtime.migration] Running upgrade 9d8a8d575284 -> cecc6bf46990, rm_time_range_endpoints_2" 1709160928670,"INFO [alembic.runtime.migration] Running upgrade cecc6bf46990 -> ad07e4fdbaba, rm_time_range_endpoints_from_qc_3" 1709160928952,"INFO [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2" 1709160929546,Traceback (most recent call last): 1709160929546," File ""/usr/local/bin/superset"", line 8, in " 1709160929546,Loaded your LOCAL configuration at [/etc/superset/superset_config.py] 1709160929546,slices updated with no time_range_endpoints: 227 1709160929546,>> Copy 7 physical tables to sl_tables... 1709160929546,>> Copy 96 SqlaTable to sl_datasets... 1709160929546, Copy dataset owners... 1709160929546, Link physical datasets with tables... 1709160929546,">> Copy 1,162 table columns to sl_columns..." 1709160929546, Link all columns to sl_datasets... 1709160929546,>> Copy 95 metrics to sl_columns... 1709160929546, Link metric columns to datasets... 1709160929546,">> Run postprocessing on 1,257 columns" 1709160929546," [Column 1 to 1,257] 1,164 may be updated" 1709160929547, sys.exit(superset()) 1709160929547," File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1157, in call" 1709160929549," return self.main(args, kwargs)" 1709160929549," File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1078, in main" 1709160929549, rv = self.invoke(ctx) 1709160929549," File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1688, in invoke" 1709160929549, return _process_result(sub_ctx.command.invoke(sub_ctx)) 1709160929549," File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1688, in invoke" 1709160929550, return _process_result(sub_ctx.command.invoke(sub_ctx)) 1709160929550," File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1434, in invoke" 1709160929550," return ctx.invoke(self.callback, ctx.params)" 1709160929550," File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 783, in invoke" 1709160929551," return __callback(args, kwargs)" 1709160929551," File ""/usr/local/lib/python3.9/site-packages/click/decorators.py"", line 33, in new_func" 1709160929551," return f(get_current_context(), *args, *kwargs)" 1709160929551," File ""/usr/local/lib/python3.9/site-packages/flask/cli.py"", line 358, in decorator" 1709160929551," return __ctx.invoke(f, args, kwargs)" 1709160929551," File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 783, in invoke" 1709160929551," return callback(*args, **kwargs)" 1709160929551," File ""/usr/local/lib/python3.9/site-packages/flask_migrate/cli.py"", line 149, in upgrade" 1709160929552," _upgrade(directory, revision, sql, tag, x_arg)" 1709160929552," File ""/usr/local/lib/python3.9/site-packages/flask_migrate/init.py"", line 98, in wrapped" 1709160929552," f(*args, kwargs)" 1709160929552," File ""/usr/local/lib/python3.9/site-packages/flask_migrate/init.py"", line 185, in upgrade" 1709160929553," command.upgrade(config, revision, sql=sql, tag=tag)" 1709160929553," File ""/usr/local/lib/python3.9/site-packages/alembic/command.py"", line 399, in upgrade" 1709160929553, script.run_env() 1709160929553," File ""/usr/local/lib/python3.9/site-packages/alembic/script/base.py"", line 578, in run_env" 1709160929553," util.load_python_file(self.dir, ""env.py"")" 1709160929553," File ""/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py"", line 93, in load_python_file" 1709160929553," module = load_module_py(module_id, path)" 1709160929553," File ""/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py"", line 109, in load_module_py" 1709160929553, spec.loader.exec_module(module) # type: ignore 1709160929553," File """", line 850, in exec_module" 1709160929553," File """", line 228, in _call_with_frames_removed" 1709160929554," File ""/usr/local/lib/python3.9/site-packages/superset/extensions/../migrations/env.py"", line 126, in " 1709160929554, run_migrations_online() 1709160929554," File ""/usr/local/lib/python3.9/site-packages/superset/extensions/../migrations/env.py"", line 118, in run_migrations_online" 1709160929554, context.run_migrations() 1709160929554," File """", line 8, in run_migrations" 1709160929554," File ""/usr/local/lib/python3.9/site-packages/alembic/runtime/environment.py"", line 937, in run_migrations" 1709160929554, self.get_context().run_migrations(kw) 1709160929554," File ""/usr/local/lib/python3.9/site-packages/alembic/runtime/migration.py"", line 624, in run_migrations" 1709160929555, step.migration_fn(**kw) 1709160929555," File ""/usr/local/lib/python3.9/site-packages/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py"", line 892, in upgrade" 1709160929555, postprocess_columns(session) 1709160929555," File ""/usr/local/lib/python3.9/site-packages/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py"", line 803, in postprocess_columns" 1709160929555, quoted_expression = get_identifier_quoter(drivername)( 1709160929555," File ""/usr/local/lib/python3.9/site-packages/superset/connectors/sqla/utils.py"", line 172, in get_identifier_quoter" 1709160929555, return SqlaURL.create(drivername).get_dialect()().identifier_preparer.quote 1709160929555," File ""/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py"", line 680, in get_dialect" 1709160929556, entrypoint = self._get_entrypoint() 1709160929556," File ""/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py"", line 662, in _get_entrypoint" 1709160929556, cls = registry.load(name) 1709160929556," File ""/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py"", line 341, in load" 1709160929556, return impl.load() 1709160929556," File ""/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/init.py"", line 208, in load" 1709160929556, module = import_module(match.group('module')) 1709160929556," File ""/usr/local/lib/python3.9/importlib/init.py"", line 127, in import_module" 1709160929556," return _bootstrap._gcd_import(name[level:], package, level)" 1709160929556," File """", line 1030, in _gcd_import" 1709160929556," File """", line 1007, in _find_and_load" 1709160929557," File """", line 986, in _find_and_load_unlocked" 1709160929557," File """", line 680, in _load_unlocked" 1709160929557," File """", line 850, in exec_module" 1709160929557," File """", line 228, in _call_with_frames_removed" 1709160929557," File ""/usr/local/lib/python3.9/site-packages/pyathena/sqlalchemy/rest.py"", line 2, in " 1709160929557, from pyathena.sqlalchemy.base import AthenaDialect 1709160929557," File ""/usr/local/lib/python3.9/site-packages/pyathena/sqlalchemy/base.py"", line 907, in " 1709160929557, class AthenaDialect(DefaultDialect): 1709160929557," File ""/usr/local/lib/python3.9/site-packages/pyathena/sqlalchemy/base.py"", line 1106, in AthenaDialect" 1709160929557, def get_view_definition( 1709160929557," File ""/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py"", line 198, in decorate" 1709160929558," decorated = _exec_code_in_env(code, env, fn.name__)" 1709160929558," File ""/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py"", line 228, in _exec_code_in_env" 1709160929558," exec(code, env)" 1709160929558," File """", line 1" 1709160929558," def get_view_definition(self, connection: 'Connection', view_name: 'str', schema: 'Optional[str]'=None, **kw) -> 'str):" 1709160929558, ^ 1709160929558,SyntaxError: EOL while scanning string literal

Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

bobelev commented 5 months ago

Same here

We use clickhouse and it has strange (but valid) queries that sometimes are not normally parsed by SQL parsers.

Can't apply new_dataset_models_take_2 migration

  File "/usr/local/lib/python3.9/site-packages/sqlglot/__init__.py", line 87, in parse
    return Dialect.get_or_raise(read or dialect).parse(sql, **opts)
  File "/usr/local/lib/python3.9/site-packages/sqlglot/dialects/dialect.py", line 490, in parse
    return self.parser(**opts).parse(self.tokenize(sql), sql)
  File "/usr/local/lib/python3.9/site-packages/sqlglot/parser.py", line 1153, in parse
    return self._parse(
  File "/usr/local/lib/python3.9/site-packages/sqlglot/parser.py", line 1222, in _parse
    self.raise_error("Invalid expression / Unexpected token")
  File "/usr/local/lib/python3.9/site-packages/sqlglot/parser.py", line 1263, in raise_error
    raise error
sqlglot.errors.ParseError: Invalid expression / Unexpected token. Line 6, Col: 30.

The above exception was the direct cause of the following exception:

  File "/usr/local/lib/python3.9/site-packages/alembic/runtime/migration.py", line 561, in run_migrations
    step.migration_fn(**kw)
  File "/app/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 895, in upgrade
    postprocess_datasets(session)
  File "/app/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 594, in postprocess_datasets
    table_refrences = extract_table_references(
  File "/app/superset/sql_parse.py", line 1037, in extract_table_references
    return parsed.tables
  File "/app/superset/sql_parse.py", line 287, in tables
    self._tables = self._extract_tables_from_sql()
  File "/app/superset/sql_parse.py", line 301, in _extract_tables_from_sql
    raise SupersetSecurityException(
superset.exceptions.SupersetSecurityException: Unable to parse SQL (generic): SELECT ts,
…redacted…

grutts commented 3 months ago

Hi do you have any updates please?

apache / superset