search

apache / superset

Apache Superset is a Data Visualization and Data Exploration Platform
https://superset.apache.org/
Apache License 2.0
61.6k stars 13.45k forks source link

Unable to upgrade from 1.5.3 to 2.0.0 or 3.1.3 #28438

Open lloonngg opened 3 months ago

lloonngg commented 3 months ago

Bug description

This issue is related to a prior one that was closed without a resolution Related issue

We are using Rancher Kubernetes 1.20.9. I don't think this is Kubernetes specific.

We were able to rebuild our Docker image including libraries to connect to various databases (pyodbc, dremio, sybase, etc.)

1.3.0 --> 1.5.0 (successful) 1.5.0 --> 1.5.3 (successful) 1.5.0 --> 2.0.0 (failed) 1.5.3 --> 2.0.0 (failed) 1.5.0 --> 3.1.3 (failed) 1.5.3 --> 3.1.3 (failed)

We are using sqlite for the database. When running superset db upgrade, we get this error below in the log. Any workarounds or solution for this?

`INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2

Copy 23 physical tables to sl_tables... Copy 899 SqlaTable to sl_datasets... Copy dataset owners... Link physical datasets with tables... Copy 8,979 table columns to sl_columns... Link all columns to sl_datasets... Copy 895 metrics to sl_columns... Link metric columns to datasets... Run postprocessing on 9,874 columns [Column 1 to 9,874] 8,929 may be updated Assign table column relations... Run postprocessing on 899 datasets Process dataset 1~899...
Traceback (most recent call last): File "/usr/local/bin/superset", line 33, in sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')()) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in call return self.main(args, kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, *kwargs) File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator return __ctx.invoke(f, args, kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade _upgrade(directory, revision, sql, tag, x_arg) File "/usr/local/lib/python3.10/site-packages/flask_migrate/init__.py", line 98, in wrapped f(*args, kwargs) File "/usr/local/lib/python3.10/site-packages/flask_migrate/init.py", line 185, in upgrade command.upgrade(config, revision, sql=sql, tag=tag) File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 294, in upgrade script.run_env() File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 490, in run_env util.load_python_file(self.dir, "env.py") File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file module = load_module_py(module_id, path) File "/usr/local/lib/python3.10/site-packages/alembic/util/compat.py", line 184, in load_module_py spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/app/superset/extensions/../migrations/env.py", line 126, in run_migrations_online() File "/app/superset/extensions/../migrations/env.py", line 118, in run_migrations_online context.run_migrations() File "", line 8, in run_migrations File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 813, in run_migrations self.get_context().run_migrations(kw) File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 561, in run_migrations step.migration_fn(*kw) File "/app/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 895, in upgrade postprocess_datasets(session) File "/app/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 594, in postprocess_datasets table_refrences = extract_table_references( File "/app/superset/sql_parse.py", line 1063, in extract_table_references return { File "/app/superset/sql_parse.py", line 1064, in Table([part["value"] for part in table["name"][::-1]]) TypeError: Table.init() takes from 2 to 4 positional arguments but 5 were given`

How to reproduce the bug

  1. Start from version 1.3.0 or 1.5.0 or 1.5.3
  2. Upgrade to 2.0.0 or 3.1.3
  3. Run superset db upgrade

Screenshots/recordings

No response

Superset version

3.1.3

Python version

3.10

Node version

I don't know

Browser

Not applicable

Additional context

No response

Checklist

dosubot[bot] commented 3 months ago

The error you're encountering, "TypeError: Table.init() takes from 2 to 4 positional arguments but 5 were given," during the superset db upgrade process is indicative of a problem with how a Table object is being initialized. This issue likely stems from a version mismatch or an incompatibility between the migration scripts and the version of SQLAlchemy being used by Superset.

To address this issue, you should:

  1. Check SQLAlchemy Version: Ensure that the version of SQLAlchemy used by Superset is compatible with the migration scripts. Incompatibilities between versions could lead to unexpected errors during migrations.

  2. Review Custom Code: If you have custom code or overrides in your Superset installation, review them to ensure they are not contributing to the issue. This includes any custom migration scripts or modifications to the Superset codebase.

  3. Enable Detailed Logging: Turning on detailed logging for the migration process can help pinpoint exactly where the error occurs, providing more context on the erroneous call to Table.__init__().

  4. Consult Documentation: Review the latest Apache Superset and SQLAlchemy documentation for any notes on migration issues or breaking changes in the versions you are upgrading to.

It's also worth noting that while SQLite is supported for Superset's metadata database, it's generally recommended for lightweight or development purposes only. For production environments, more robust databases like PostgreSQL or MySQL are recommended due to their better scalability and concurrency support.

#### Details **Sources** - [Error while upgrading superset to 2.0.0 psycopg2.errors.UndefinedColumn](https://github.com/apache/superset/issues/22180) - [superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py](https://github.com/apache/superset/blob/master/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py) - [docs/docs/faq.mdx](https://github.com/apache/superset/blob/master/docs/docs/faq.mdx) - [docs/docs/configuration/databases.mdx](https://github.com/apache/superset/blob/master/docs/docs/configuration/databases.mdx) - [CHANGELOG/2.0.0.md](https://github.com/apache/superset/blob/master/CHANGELOG/2.0.0.md) - [RELEASING/release-notes-2-0/changelog.md](https://github.com/apache/superset/blob/master/RELEASING/release-notes-2-0/changelog.md) - [UPDATING.md](https://github.com/apache/superset/blob/master/UPDATING.md) - [CHANGELOG/3.0.0.md](https://github.com/apache/superset/blob/master/CHANGELOG/3.0.0.md)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.