I'm getting ERROR:flask_migrate:Error while upgrading.

lei1402 commented 1 week ago

Bug description

i use python 3.10.9 superset 2.0.0 --> 4.0.2

How to reproduce the bug

I tried removing .superset and then doing superset db init but still got the same error. After I removed the directory in the path, it still required the directory. I don't know what data to delete before executing.

superset db upgrade superset db init logging was configured successfully 2024-09-10 20:21:45,912:INFO:superset.utils.logging_configurator:logging was configured successfully 2024-09-10 20:21:45,916:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'> /home/superset/anaconda3/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( 2024-09-10 20:21:46,856:ERROR:flask_migrate:Error: Directory /home/superset/anaconda3/lib/python3.10/site-packages/superset/extensions/../migrations already exists and is not empty

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.10

Node version

16

Browser

Chrome

Additional context

No response

Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

dosubot[bot] commented 1 week ago

Hey there, @lei1402! I'm here to help you with any bugs, questions, or contributions you have for Superset. Let's tackle this challenge together!

I found a similar issue that might help you resolve the "ERROR:flask_migrate:Error while upgrading" when upgrading Superset from version 2.0.0 to 4.0.2 using Python 3.10.9. Here are the steps to follow:

Shut down the running containers (if using Docker Compose):
```
docker compose down
```
Update the Superset repository:
```
git pull origin master
```
Restart the containers (if using Docker Compose):
```
docker compose up
```
Upgrade Superset manually (if not using Docker Compose):
```
pip install apache-superset --upgrade
```
Upgrade the metadata database:
```
superset db upgrade
superset init
```

Ensure you have backed up your metadata database before performing these steps ^[1].

Additionally, the warning about using in-memory storage for Flask-Limiter indicates that you should configure a proper storage backend for rate limiting. You can configure Redis as the storage backend by setting the RATELIMIT_STORAGE_URI in the superset/config.py file:

RATELIMIT_STORAGE_URI = "redis://host:port"

Replace host and port with the appropriate values for your Redis instance ^[2].

If you encounter any issues with the migration scripts or directories, you might need to check the specific migration scripts involved in the upgrade process. However, I couldn't find detailed information on the specific migration scripts or directories involved in the Superset upgrade process from version 2.0.0 to 4.0.2 ^[1].

To continue talking to Dosu, mention @dosu.

lei1402 commented 1 week ago

superset db upgrade superset init Superset db upgrade was successful, but an error occurred during init @dosu

dosubot[bot] commented 1 week ago

The error during superset init after a successful superset db upgrade is likely due to an issue in the migration script 4a33124c18ad_mig_new_columnar_upload_perm.py. Specifically, the error handling in the upgrade function might not be sufficient to catch and handle all potential issues that could arise during the permission migration process.

Here is the relevant part of the migration script:

def upgrade():
    bind = op.get_bind()
    session = Session(bind=bind)

    do_upgrade(session)

    try:
        session.commit()
    except SQLAlchemyError as ex:
        session.rollback()
        raise Exception(f"An error occurred while upgrading permissions: {ex}") from ex

If an error occurs during the do_upgrade(session) call, it will be caught by the except block, which rolls back the session and raises a new exception with a message indicating that an error occurred while upgrading permissions. This could be the source of the error you encountered during superset init.

To diagnose the specific cause, you would need to check the exact error message that was raised. This message should provide more details about what went wrong during the permission migration process ^[1].

To continue talking to Dosu, mention @dosu.

lei1402 commented 1 week ago

superset db upgrade logging was configured successfully 2024-09-10 20:57:54,361:INFO:superset.utils.logging_configurator:logging was configured successfully 2024-09-10 20:57:54,365:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'> /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( WARNI [alembic.env] SQLite Database support for metadata databases will be removed in a future version of Superset. INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade e09b4ae78457 -> f3afaf1f11f0, add_unique_name_desc_rls INFO [alembic.runtime.migration] Running upgrade f3afaf1f11f0 -> 7fb8bca906d2, permalink_rename_filterState INFO [alembic.runtime.migration] Running upgrade 7fb8bca906d2 -> cdcf3d64daf4, Add user_id and dttm composite index to Log model INFO [alembic.runtime.migration] Running upgrade cdcf3d64daf4 -> c747c78868b6, Migrating legacy TreeMap INFO [alembic.runtime.migration] Running upgrade c747c78868b6 -> 06e1e70058c7, Migrating legacy Area Upgraded 8/8 charts INFO [alembic.runtime.migration] Running upgrade 06e1e70058c7 -> a39867932713, query_context_to_mediumtext INFO [alembic.runtime.migration] Running upgrade a39867932713 -> 409c7b420ab0, add created_by_fk as owner INFO [alembic.runtime.migration] Running upgrade 409c7b420ab0 -> ffa79af61a56, rename report_schedule.extra to extra_json INFO [alembic.runtime.migration] Running upgrade ffa79af61a56 -> 6d3c6f9d665d, fix_table_chart_conditional_formatting_colors INFO [alembic.runtime.migration] Running upgrade 6d3c6f9d665d -> 291f024254b5, drop_column_allow_multi_schema_metadata_fetch INFO [alembic.runtime.migration] Running upgrade 291f024254b5 -> deb4c9d4a4ef, parameters in saved queries INFO [alembic.runtime.migration] Running upgrade deb4c9d4a4ef -> 4ce1d9b25135, remove_filter_bar_orientation INFO [alembic.runtime.migration] Running upgrade 4ce1d9b25135 -> f3c2d8ec8595, create_ssh_tunnel_credentials_tbl INFO [alembic.runtime.migration] Running upgrade f3c2d8ec8595 -> 9c2a5681ddfd, convert key-value entries to json INFO [alembic.runtime.migration] Running upgrade 9c2a5681ddfd -> c0a3ea245b61, remove_show_native_filters INFO [alembic.runtime.migration] Running upgrade c0a3ea245b61 -> d0ac08bb5b83, invert_horizontal_bar_chart_order INFO [alembic.runtime.migration] Running upgrade d0ac08bb5b83 -> b5ea9d343307, bar_chart_stack_options INFO [alembic.runtime.migration] Running upgrade b5ea9d343307 -> 07f9a902af1b, drop postgres enum constrains for tags INFO [alembic.runtime.migration] Running upgrade 07f9a902af1b -> 7e67aecbf3f1, chart-ds-constraint INFO [alembic.runtime.migration] Running upgrade 7e67aecbf3f1 -> 4ea966691069, cross-filter-global-scoping INFO [alembic.runtime.migration] Running upgrade 4ea966691069 -> 9ba2ce3086e5, migrate-pivot-table-v1-to-v2 INFO [alembic.runtime.migration] Running upgrade 9ba2ce3086e5 -> 4c5da39be729, migrate_treemap_chart INFO [alembic.runtime.migration] Running upgrade 4c5da39be729 -> ae58e1e58e5c, migrate_dual_line_to_mixed_chart INFO [alembic.runtime.migration] Running upgrade ae58e1e58e5c -> 83e1abbe777f, drop access_request INFO [alembic.runtime.migration] Running upgrade 83e1abbe777f -> 90139bf715e4, add_currency_column_to_metrics INFO [alembic.runtime.migration] Running upgrade 90139bf715e4 -> 6fbe660cac39, add on delete cascade for tables references INFO [alembic.runtime.migration] Running upgrade 6fbe660cac39 -> 8e5b0fb85b9a, Add custom size columns to report schedule INFO [alembic.runtime.migration] Running upgrade 8e5b0fb85b9a -> 240d23c7f86f, update_tag_model_w_description INFO [alembic.runtime.migration] Running upgrade 240d23c7f86f -> f92a3124dd66, drop rouge constraints and tables INFO [alembic.runtime.migration] Running upgrade f92a3124dd66 -> 6d05b0a70c89, add on delete cascade for owners references INFO [alembic.runtime.migration] Running upgrade 6d05b0a70c89 -> 863adcf72773, delete obsolete Druid NoSQL slice parameters INFO [alembic.runtime.migration] Running upgrade 863adcf72773 -> a23c6f8b1280, cleanup erroneous parent filter IDs INFO [alembic.runtime.migration] Running upgrade a23c6f8b1280 -> bf646a0c1501, json_metadata INFO [alembic.runtime.migration] Running upgrade bf646a0c1501 -> e0f6f91c2055, create_user_favorite_table INFO [alembic.runtime.migration] Running upgrade e0f6f91c2055 -> ee179a490af9, deckgl-path-width-units INFO [alembic.runtime.migration] Running upgrade ee179a490af9 -> 0769ef90fddd, Fix schema perm for datasets INFO [alembic.runtime.migration] Running upgrade 0769ef90fddd -> 2e826adca42c, Fix schema for log INFO [alembic.runtime.migration] Running upgrade 2e826adca42c -> 8ace289026f3, add on delete cascade for dashboard_slices INFO [alembic.runtime.migration] Running upgrade 8ace289026f3 -> 4448fa6deeb1, add on delete cascade for embedded_dashboards INFO [alembic.runtime.migration] Running upgrade 4448fa6deeb1 -> 9f4a086c2676, add_normalize_columns_to_sqla_model INFO [alembic.runtime.migration] Running upgrade 9f4a086c2676 -> ec54aca4c8a2, Increase ab_user.email field size INFO [alembic.runtime.migration] Running upgrade ec54aca4c8a2 -> 317970b4400c, Added always_filter_main_dttm to datasource INFO [alembic.runtime.migration] Running upgrade 317970b4400c -> 4b85906e5b91, add on delete cascade for dashboard_roles INFO [alembic.runtime.migration] Running upgrade 4b85906e5b91 -> b7851ee5522f, replay 317970b4400c INFO [alembic.runtime.migration] Running upgrade b7851ee5522f -> 06dd9ff00fe8, add_percent_calculation_type_funnel_chart INFO [alembic.runtime.migration] Running upgrade 06dd9ff00fe8 -> 65a167d4c62e, add indexes to report models INFO [alembic.runtime.migration] Running upgrade 65a167d4c62e -> 59a1450b3c10, drop_filter_sets_table INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> a32e0c4d8646, migrate-sunburst-chart INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> 96164e3017c6 /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations/versions/2024-01-17_13-09_96164e3017c6_tagged_object_unique_constraint.py:58: SAWarning: WARNING: SQL-parsed foreign key constraint '('database_id', 'dbs', 'id')' could not be located in PRAGMA foreign_keys for table tab_state metadata.reflect(bind=bind) /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations/versions/2024-01-17_13-09_96164e3017c6_tagged_object_unique_constraint.py:58: SAWarning: WARNING: SQL-parsed foreign key constraint '('latest_query_id', 'query', 'client_id')' could not be located in PRAGMA foreign_keys for table tab_state metadata.reflect(bind=bind) /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations/versions/2024-01-17_13-09_96164e3017c6_tagged_object_unique_constraint.py:58: SAWarning: WARNING: SQL-parsed foreign key constraint '('database_id', 'dbs', 'id')' could not be located in PRAGMA foreign_keys for table table_schema metadata.reflect(bind=bind) INFO [alembic.runtime.migration] Running upgrade 96164e3017c6, a32e0c4d8646 -> 15a2c68a2e6b, merging two heads INFO [alembic.runtime.migration] Running upgrade a32e0c4d8646 -> 214f580d09c9, migrate_filter_boxes_to_native_filters INFO [alembic.runtime.migration] Running upgrade 214f580d09c9 -> e863403c0c50, drop_url_table INFO [alembic.runtime.migration] Running upgrade e863403c0c50, 15a2c68a2e6b -> 1cf8e4344e2b, merging INFO [alembic.runtime.migration] Running upgrade 1cf8e4344e2b -> 87d38ad83218, Migrate can_view_and_drill permission INFO [alembic.runtime.migration] Running upgrade 87d38ad83218 -> 17fcea065655, change_text_to_mediumtext (base) [superset@superset ~]$ superset db init logging was configured successfully 2024-09-10 20:58:41,123:INFO:superset.utils.logging_configurator:logging was configured successfully 2024-09-10 20:58:41,126:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'> /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( 2024-09-10 20:58:42,031:ERROR:flask_migrate:Error: Directory /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/extensions/../migrations already exists and is not empty @dosu Still the same error

lei1402 commented 1 week ago

@dosu Ignore the speed limit first and let him go through init first.

dosubot[bot] commented 1 week ago

The error you're encountering, "Directory already exists and is not empty," indicates that the /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations directory already exists and contains files. This can cause issues during the superset db init process.

To resolve this, you can try the following steps:

Backup the existing migrations directory:

mv /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations_backup

Recreate the migrations directory:

mkdir /home/superset/anaconda3/envs/superset/lib/python3.9/site-packages/superset/migrations

Run the superset db init command again:
```
superset db init
```

This should allow the superset db init process to proceed without encountering the existing directory issue.

Additionally, to address the in-memory storage warning for Flask-Limiter, you can configure a proper storage backend like Redis. Add the following configuration to your superset/config.py file:

RATELIMIT_STORAGE_URI = "redis://localhost:6379"

Replace localhost:6379 with the appropriate values for your Redis instance ^[1].

To continue talking to Dosu, mention @dosu.

apache / superset