search

databrickslabs / ucx

Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Other
193 stars 69 forks source link

whitelist dbx #2001

Closed ericvergnaud closed 18 hours ago

ericvergnaud commented 2 days ago

Changes

whitelist dbx

Linked issues

Progresses #1901

Functionality

None

Tests

github-actions[bot] commented 2 days ago

❌ 94/100 passed, 6 failed, 22 skipped, 5h22m26s total

❌ test_create_catalog_schema_with_principal_acl_CLOUD_ENV: databricks.sdk.errors.platform.BadRequest: [RequestId=527d3f3c-577c-4f7d-a00a-525744cc0d0e ErrorClass=PRINCIPAL_DOES_NOT_EXIST] Could not find principal with name ucx_Gjwse (4m50.341s) ``` databricks.sdk.errors.platform.BadRequest: [RequestId=527d3f3c-577c-4f7d-a00a-525744cc0d0e ErrorClass=PRINCIPAL_DOES_NOT_EXIST] Could not find principal with name ucx_Gjwse 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr 09:36 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sinf5: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sinf5 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sinf5.ucx_txnqi: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sinf5/ucx_txnqi 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sinf5.ucx_txnqi', metastore_id=None, name='ucx_txnqi', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sinf5', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999412604, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clk8a', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999412604, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:37 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_clk8a.ucx_sinf5: https://DATABRICKS_HOST/explore/data/ucx_clk8a/ucx_sinf5 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clk8a', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr 09:36 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sinf5: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sinf5 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sinf5.ucx_txnqi: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sinf5/ucx_txnqi 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sinf5.ucx_txnqi', metastore_id=None, name='ucx_txnqi', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sinf5', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999412604, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clk8a', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999412604, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:37 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_clk8a.ucx_sinf5: https://DATABRICKS_HOST/explore/data/ucx_clk8a/ucx_sinf5 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clk8a', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-eow1kvyi@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-eow1kvyi@example.com')], entitlements=[], external_id=None, groups=[], id='6537728990542218', name=Name(family_name=None, given_name='sdk-eow1kvyi@example.com'), roles=[], schemas=[, ], user_name='sdk-eow1kvyi@example.com') 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0703-093702-lct13jr [group_name admins CAN_MANAGE] -> [user_name sdk-eow1kvyi@example.com CAN_ATTACH_TO] 09:37 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_spqbr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spqbr 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spqbr', metastore_id=None, name='ucx_spqbr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas 09:37 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Preparing a list of UC catalogs and schema to be created 09:37 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_spqbr.tables] fetching tables inventory 09:37 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_spqbr.tables] crawling new batch for tables 09:37 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sinf5] listing tables 09:38 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sinf5.ucx_txnqi] fetching table metadata 09:38 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_spqbr.tables] found 1 new records for tables 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `ucx_GsHFt` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `serge.smertin@databricks.com` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `sdk-jciqy45k@example.com` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `ucx_Gjwse` 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr 09:36 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sinf5: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sinf5 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sinf5.ucx_txnqi: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sinf5/ucx_txnqi 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sinf5.ucx_txnqi', metastore_id=None, name='ucx_txnqi', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sinf5', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999412604, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clk8a', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999412604, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:37 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_clk8a.ucx_sinf5: https://DATABRICKS_HOST/explore/data/ucx_clk8a/ucx_sinf5 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clk8a', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-eow1kvyi@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-eow1kvyi@example.com')], entitlements=[], external_id=None, groups=[], id='6537728990542218', name=Name(family_name=None, given_name='sdk-eow1kvyi@example.com'), roles=[], schemas=[, ], user_name='sdk-eow1kvyi@example.com') 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0703-093702-lct13jr [group_name admins CAN_MANAGE] -> [user_name sdk-eow1kvyi@example.com CAN_ATTACH_TO] 09:37 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_spqbr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spqbr 09:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spqbr', metastore_id=None, name='ucx_spqbr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:37 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas 09:37 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Preparing a list of UC catalogs and schema to be created 09:37 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_spqbr.tables] fetching tables inventory 09:37 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_spqbr.tables] crawling new batch for tables 09:37 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sinf5] listing tables 09:38 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sinf5.ucx_txnqi] fetching table metadata 09:38 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_spqbr.tables] found 1 new records for tables 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `ucx_GsHFt` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `serge.smertin@databricks.com` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `sdk-jciqy45k@example.com` 09:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_clk8a.ucx_sinf5') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_clk8a.ucx_sinf5 TO `ucx_Gjwse` 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster permissions fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster permissions fixture: 0703-093702-lct13jr [group_name admins CAN_MANAGE] -> [user_name sdk-eow1kvyi@example.com CAN_ATTACH_TO] 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999412604, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clk8a', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999412604, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster fixture: 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 table fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sinf5.ucx_txnqi', metastore_id=None, name='ucx_txnqi', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sinf5', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/tMKr', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clk8a', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clk8a.ucx_sinf5', metastore_id=None, name='ucx_sinf5', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:40 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spqbr', metastore_id=None, name='ucx_spqbr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures 09:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-eow1kvyi@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-eow1kvyi@example.com')], entitlements=[], external_id=None, groups=[], id='6537728990542218', name=Name(family_name=None, given_name='sdk-eow1kvyi@example.com'), roles=[], schemas=[, ], user_name='sdk-eow1kvyi@example.com') [gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_move_views: databricks.sdk.errors.platform.BadRequest: (4m8.141s) ``` databricks.sdk.errors.platform.BadRequest: [PARSE_SYNTAX_ERROR] Syntax error at or near '.6832'. SQLSTATE: 42601 (line 1, pos 21) == SQL == CREATE VIEW ucx_cdt0d.6832.ucx_tzsfv AS select * from ucx_cbnec.ucx_sez4l.ucx_tcxno ---------------------^^^ [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999882646, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbnec', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cbnec', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999882646, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cbnec.ucx_sez4l: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cbnec', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbnec.ucx_sez4l', metastore_id=None, name='ucx_sez4l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Table ucx_cbnec.ucx_sez4l.ucx_tcxno: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l/ucx_tcxno 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_tcxno', metastore_id=None, name='ucx_tcxno', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_sez4l/ucx_tcxno', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:45 INFO [databricks.labs.ucx.mixins.fixtures] Table ucx_cbnec.ucx_sez4l.ucx_tzsfv: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l/ucx_tzsfv 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_tzsfv', metastore_id=None, name='ucx_tzsfv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) 09:46 INFO [databricks.labs.ucx.mixins.fixtures] Table ucx_cbnec.ucx_sez4l.ucx_trseg: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l/ucx_trseg 09:46 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_trseg', metastore_id=None, name='ucx_trseg', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) 09:46 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000007805, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdt0d', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdt0d', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000007805, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:46 WARNING [databricks.labs.ucx.hive_metastore.table_migrate] schema 6832 not found in ucx_cdt0d, creating... 09:46 INFO [databricks.labs.ucx.hive_metastore.table_migrate] Moved 0 tables to the new schema 6832. 09:46 INFO [databricks.labs.ucx.hive_metastore.table_migrate] Creating view ucx_cdt0d.6832.ucx_tzsfv 09:47 ERROR [databricks.labs.blueprint.parallel] Creating views('ucx_cbnec', 'ucx_sez4l', 'ucx_tzsfv', 'ucx_cdt0d', '6832', del_view=True, view_text='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno') task failed: [PARSE_SYNTAX_ERROR] Syntax error at or near '.6832'. SQLSTATE: 42601 (line 1, pos 21) == SQL == CREATE VIEW ucx_cdt0d.6832.ucx_tzsfv AS select * from ucx_cbnec.ucx_sez4l.ucx_tcxno ---------------------^^^ Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_move.py", line 276, in _move_view self._recreate_view(to_view_name, view_text) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_move.py", line 293, in _recreate_view self._execute(create_sql) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 142, in execute self._sql.execute(sql, catalog=catalog, schema=schema) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 265, in execute self._raise_if_needed(result_status) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 479, in _raise_if_needed raise error_class(error_message) databricks.sdk.errors.platform.BadRequest: [PARSE_SYNTAX_ERROR] Syntax error at or near '.6832'. SQLSTATE: 42601 (line 1, pos 21) == SQL == CREATE VIEW ucx_cdt0d.6832.ucx_tzsfv AS select * from ucx_cbnec.ucx_sez4l.ucx_tcxno ---------------------^^^ 09:47 CRITICAL [databricks.labs.blueprint.parallel] All 'Creating views' tasks failed!!! 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999882646, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbnec', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cbnec', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999882646, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cbnec.ucx_sez4l: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cbnec', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbnec.ucx_sez4l', metastore_id=None, name='ucx_sez4l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Table ucx_cbnec.ucx_sez4l.ucx_tcxno: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l/ucx_tcxno 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_tcxno', metastore_id=None, name='ucx_tcxno', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_sez4l/ucx_tcxno', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:45 INFO [databricks.labs.ucx.mixins.fixtures] Table ucx_cbnec.ucx_sez4l.ucx_tzsfv: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l/ucx_tzsfv 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_tzsfv', metastore_id=None, name='ucx_tzsfv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) 09:46 INFO [databricks.labs.ucx.mixins.fixtures] Table ucx_cbnec.ucx_sez4l.ucx_trseg: https://DATABRICKS_HOST/explore/data/ucx_cbnec/ucx_sez4l/ucx_trseg 09:46 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_trseg', metastore_id=None, name='ucx_trseg', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) 09:46 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000007805, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdt0d', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdt0d', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000007805, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:46 WARNING [databricks.labs.ucx.hive_metastore.table_migrate] schema 6832 not found in ucx_cdt0d, creating... 09:46 INFO [databricks.labs.ucx.hive_metastore.table_migrate] Moved 0 tables to the new schema 6832. 09:46 INFO [databricks.labs.ucx.hive_metastore.table_migrate] Creating view ucx_cdt0d.6832.ucx_tzsfv 09:47 ERROR [databricks.labs.blueprint.parallel] Creating views('ucx_cbnec', 'ucx_sez4l', 'ucx_tzsfv', 'ucx_cdt0d', '6832', del_view=True, view_text='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno') task failed: [PARSE_SYNTAX_ERROR] Syntax error at or near '.6832'. SQLSTATE: 42601 (line 1, pos 21) == SQL == CREATE VIEW ucx_cdt0d.6832.ucx_tzsfv AS select * from ucx_cbnec.ucx_sez4l.ucx_tcxno ---------------------^^^ Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_move.py", line 276, in _move_view self._recreate_view(to_view_name, view_text) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_move.py", line 293, in _recreate_view self._execute(create_sql) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 142, in execute self._sql.execute(sql, catalog=catalog, schema=schema) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 265, in execute self._raise_if_needed(result_status) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 479, in _raise_if_needed raise error_class(error_message) databricks.sdk.errors.platform.BadRequest: [PARSE_SYNTAX_ERROR] Syntax error at or near '.6832'. SQLSTATE: 42601 (line 1, pos 21) == SQL == CREATE VIEW ucx_cdt0d.6832.ucx_tzsfv AS select * from ucx_cbnec.ucx_sez4l.ucx_tcxno ---------------------^^^ 09:47 CRITICAL [databricks.labs.blueprint.parallel] All 'Creating views' tasks failed!!! 09:47 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 09:47 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_tcxno', metastore_id=None, name='ucx_tcxno', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_sez4l/ucx_tcxno', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:47 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_tzsfv', metastore_id=None, name='ucx_tzsfv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_tzsfv', metastore_id=None, name='ucx_tzsfv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But ucx_cbnec.ucx_sez4l.ucx_tzsfv is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_trseg', metastore_id=None, name='ucx_trseg', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='ucx_cbnec', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='ucx_cbnec.ucx_sez4l.ucx_trseg', metastore_id=None, name='ucx_trseg', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_sez4l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='select * from ucx_cbnec.ucx_sez4l.ucx_tcxno', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But ucx_cbnec.ucx_sez4l.ucx_trseg is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cbnec', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbnec.ucx_sez4l', metastore_id=None, name='ucx_sez4l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 2 catalog fixtures 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999882646, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbnec', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cbnec', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999882646, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000007805, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdt0d', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdt0d', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000007805, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: TimeoutError: Timed out after 0:05:00 (14m2.723s) ``` TimeoutError: Timed out after 0:05:00 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/uGw4 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.parquet_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/parquet_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.parquet_serde_iv6io', metastore_id=None, name='parquet_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/parquet_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.orc_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/orc_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.orc_serde_iv6io', metastore_id=None, name='orc_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/orc_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.avro_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/avro_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.avro_serde_iv6io', metastore_id=None, name='avro_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/avro_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999844239, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cnize', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999844239, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cnize.hiveserde_in_place_iv6io: https://DATABRICKS_HOST/explore/data/ucx_cnize/hiveserde_in_place_iv6io 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cnize', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sccpd: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sccpd 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sccpd', metastore_id=None, name='ucx_sccpd', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/uGw4 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.parquet_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/parquet_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.parquet_serde_iv6io', metastore_id=None, name='parquet_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/parquet_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.orc_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/orc_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.orc_serde_iv6io', metastore_id=None, name='orc_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/orc_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.avro_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/avro_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.avro_serde_iv6io', metastore_id=None, name='avro_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/avro_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999844239, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cnize', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999844239, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cnize.hiveserde_in_place_iv6io: https://DATABRICKS_HOST/explore/data/ucx_cnize/hiveserde_in_place_iv6io 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cnize', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sccpd: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sccpd 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sccpd', metastore_id=None, name='ucx_sccpd', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:47 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.7Iby/config.yml) doesn't exist. 09:47 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:47 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:47 INFO [databricks.labs.ucx.install] Fetching installations... 09:47 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.7Iby is corrupted. Skipping... 09:47 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:47 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:47 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:47 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7220240703094713 09:47 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:50 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.7Iby/README for the next steps. 09:50 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/1066837654065914 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/uGw4 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.parquet_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/parquet_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.parquet_serde_iv6io', metastore_id=None, name='parquet_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/parquet_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.orc_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/orc_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.orc_serde_iv6io', metastore_id=None, name='orc_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/orc_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_iv6io.avro_serde_iv6io: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_iv6io/avro_serde_iv6io 09:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.avro_serde_iv6io', metastore_id=None, name='avro_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/avro_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999844239, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cnize', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999844239, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cnize.hiveserde_in_place_iv6io: https://DATABRICKS_HOST/explore/data/ucx_cnize/hiveserde_in_place_iv6io 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cnize', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sccpd: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sccpd 09:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sccpd', metastore_id=None, name='ucx_sccpd', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:47 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.7Iby/config.yml) doesn't exist. 09:47 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:47 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:47 INFO [databricks.labs.ucx.install] Fetching installations... 09:47 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.7Iby is corrupted. Skipping... 09:47 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:47 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:47 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:47 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7220240703094713 09:47 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:50 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:50 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.7Iby/README for the next steps. 09:50 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/1066837654065914 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_storage_dir fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/uGw4 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719999844239, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cnize', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719999844239, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:54 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7220240703094713 from https://DATABRICKS_HOST 09:54 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sccpd 09:54 INFO [databricks.labs.ucx.install] Deleting jobs 09:54 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=986426050320138. 09:54 INFO [databricks.labs.ucx.install] Deleting assessment job_id=983486476792157. 09:55 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=253476164315690. 09:55 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=444940692370794. 09:55 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=801981107743955. 09:55 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=110804384993477. 09:55 INFO [databricks.labs.ucx.install] Deleting failing job_id=924062200788405. 09:55 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=584475324854955. 09:55 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=796549310351843. 09:55 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=349722348446582. 09:55 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=596044124141371. 09:55 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=516438119062225. 09:55 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=1066837654065914. 09:55 INFO [databricks.labs.ucx.install] Deleting cluster policy 09:55 INFO [databricks.labs.ucx.install] Deleting secret scope 09:55 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.parquet_serde_iv6io', metastore_id=None, name='parquet_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/parquet_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.orc_serde_iv6io', metastore_id=None, name='orc_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/orc_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_iv6io.avro_serde_iv6io', metastore_id=None, name='avro_serde_iv6io', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070310'}, row_filter=None, schema_name='hiveserde_in_place_iv6io', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_iv6io/avro_serde_iv6io', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cnize', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cnize.hiveserde_in_place_iv6io', metastore_id=None, name='hiveserde_in_place_iv6io', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sccpd', metastore_id=None, name='ucx_sccpd', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_migrate_view: databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92d6a658-e942-4039-9896-2158d727b0ad.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. (16m5.733s) ``` databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92d6a658-e942-4039-9896-2158d727b0ad.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720001076648,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1723","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"4","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"4","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"751999cb-021e-489c-81cb-926db318d79d"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. [gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:53 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_spbzh: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh 09:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spbzh', metastore_id=None, name='ucx_spbzh', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:53 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spbzh.ucx_tcvxy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh/ucx_tcvxy 09:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_tcvxy', metastore_id=None, name='ucx_tcvxy', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_spbzh/ucx_tcvxy', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:54 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spbzh.ucx_txhx9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh/ucx_txhx9 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_txhx9', metastore_id=None, name='ucx_txhx9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_tcvxy', view_dependencies=None) 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spbzh.ucx_t1suw: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh/ucx_t1suw 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_t1suw', metastore_id=None, name='ucx_t1suw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_txhx9', view_dependencies=None) 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000575130, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cqm4y', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cqm4y', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000575130, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cqm4y.ucx_spbzh: https://DATABRICKS_HOST/explore/data/ucx_cqm4y/ucx_spbzh 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cqm4y', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cqm4y.ucx_spbzh', metastore_id=None, name='ucx_spbzh', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:56 INFO [tests.integration.hive_metastore.test_migrate] dst_catalog=ucx_cqm4y, managed_table=hive_metastore.ucx_spbzh.ucx_tcvxy 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_swevs: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swevs 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swevs', metastore_id=None, name='ucx_swevs', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 09:57 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] crawling new batch for migration_status 09:57 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 09:57 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] crawling new batch for tables 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh] listing tables 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.ucx_t1suw] fetching table metadata 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.ucx_tcvxy] fetching table metadata 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.ucx_txhx9] fetching table metadata 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.view3] fetching table metadata 09:58 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] found 4 new records for tables 09:58 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] found 4 new records for migration_status 09:59 WARNING [databricks.labs.ucx.hive_metastore.migration_status] Schema ucx_cm82l.ucx_s63gm no longer exists. Skipping checking its migration status. 09:59 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:00 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating managed table hive_metastore.ucx_spbzh.ucx_tcvxy to ucx_cqm4y.ucx_spbzh.ucx_tcvxy using SQL query: CREATE TABLE IF NOT EXISTS ucx_cqm4y.ucx_spbzh.ucx_tcvxy DEEP CLONE hive_metastore.ucx_spbzh.ucx_tcvxy; 10:01 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:02 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:02 INFO [databricks.labs.ucx.hive_metastore.mapping] The intended target for hive_metastore.ucx_spbzh.ucx_tcvxy, ucx_cqm4y.ucx_spbzh.ucx_tcvxy, already exists. 10:03 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:03 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] crawling new batch for migration_status 10:03 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:03 WARNING [databricks.labs.ucx.hive_metastore.migration_status] Catalog ucx_cscrh no longer exists. Skipping checking its migration status. 10:04 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spbzh.ucx_tcvxy is set as migrated 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] found 4 new records for migration_status 10:04 ERROR [databricks.labs.blueprint.parallel] migrate views(ViewToMigrate(src=Table(catalog='hive_metastore', database='ucx_spbzh', name='view3', object_type='VIEW', table_format='UNKNOWN', location=None, view_text='SELECT * FROM hive_metastore.ucx_spbzh.ucx_tcvxy', upgraded_to=None, storage_properties='{serialization.format: 1}', is_partitioned=False), rule=Rule(workspace_name='workspace', catalog_name='ucx_cqm4y', src_schema='ucx_spbzh', dst_schema='ucx_spbzh', src_table='view3', dst_table='view3', recon_tolerance_percent=0, compare_rows=False)), []) task failed: This transaction attempted to read one or more files that were deleted (for example part-00000-92d6a658-e942-4039-9896-2158d727b0ad.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720001076648,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1723","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"4","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"4","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"751999cb-021e-489c-81cb-926db318d79d"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 200, in _migrate_view if self._view_can_be_migrated(src_view): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 208, in _view_can_be_migrated if not self.index_full_refresh().get(table.schema, table.name): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 65, in index_full_refresh self._migration_status_refresher.reset() File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 55, in reset self._exec(f"DELETE FROM {self.full_name}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 142, in execute self._sql.execute(sql, catalog=catalog, schema=schema) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 265, in execute self._raise_if_needed(result_status) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 479, in _raise_if_needed raise error_class(error_message) databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92d6a658-e942-4039-9896-2158d727b0ad.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720001076648,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1723","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"4","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"4","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"751999cb-021e-489c-81cb-926db318d79d"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] crawling new batch for migration_status 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:05 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spbzh.ucx_tcvxy is set as migrated 10:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] found 4 new records for migration_status 10:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:06 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating view hive_metastore.ucx_spbzh.ucx_txhx9 to using SQL query: CREATE VIEW IF NOT EXISTS ucx_cqm4y.ucx_spbzh.ucx_txhx9 (id, value) TBLPROPERTIES ('RemoveAfter'='2024070311', 'transient_lastDdlTime'='1720000471') AS SELECT * FROM ucx_cqm4y.ucx_spbzh.ucx_tcvxy 10:06 ERROR [databricks.labs.blueprint.parallel] More than half 'migrate views' tasks failed: 50% results available (1/2). Took 0:02:26.078733 09:53 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_spbzh: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh 09:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spbzh', metastore_id=None, name='ucx_spbzh', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:53 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spbzh.ucx_tcvxy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh/ucx_tcvxy 09:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_tcvxy', metastore_id=None, name='ucx_tcvxy', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_spbzh/ucx_tcvxy', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:54 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spbzh.ucx_txhx9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh/ucx_txhx9 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_txhx9', metastore_id=None, name='ucx_txhx9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_tcvxy', view_dependencies=None) 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spbzh.ucx_t1suw: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spbzh/ucx_t1suw 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_t1suw', metastore_id=None, name='ucx_t1suw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_txhx9', view_dependencies=None) 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000575130, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cqm4y', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cqm4y', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000575130, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cqm4y.ucx_spbzh: https://DATABRICKS_HOST/explore/data/ucx_cqm4y/ucx_spbzh 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cqm4y', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cqm4y.ucx_spbzh', metastore_id=None, name='ucx_spbzh', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:56 INFO [tests.integration.hive_metastore.test_migrate] dst_catalog=ucx_cqm4y, managed_table=hive_metastore.ucx_spbzh.ucx_tcvxy 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_swevs: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swevs 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swevs', metastore_id=None, name='ucx_swevs', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 09:56 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 09:57 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] crawling new batch for migration_status 09:57 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 09:57 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] crawling new batch for tables 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh] listing tables 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.ucx_t1suw] fetching table metadata 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.ucx_tcvxy] fetching table metadata 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.ucx_txhx9] fetching table metadata 09:57 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spbzh.view3] fetching table metadata 09:58 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] found 4 new records for tables 09:58 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] found 4 new records for migration_status 09:59 WARNING [databricks.labs.ucx.hive_metastore.migration_status] Schema ucx_cm82l.ucx_s63gm no longer exists. Skipping checking its migration status. 09:59 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:00 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating managed table hive_metastore.ucx_spbzh.ucx_tcvxy to ucx_cqm4y.ucx_spbzh.ucx_tcvxy using SQL query: CREATE TABLE IF NOT EXISTS ucx_cqm4y.ucx_spbzh.ucx_tcvxy DEEP CLONE hive_metastore.ucx_spbzh.ucx_tcvxy; 10:01 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:02 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:02 INFO [databricks.labs.ucx.hive_metastore.mapping] The intended target for hive_metastore.ucx_spbzh.ucx_tcvxy, ucx_cqm4y.ucx_spbzh.ucx_tcvxy, already exists. 10:03 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:03 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] crawling new batch for migration_status 10:03 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:03 WARNING [databricks.labs.ucx.hive_metastore.migration_status] Catalog ucx_cscrh no longer exists. Skipping checking its migration status. 10:04 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spbzh.ucx_tcvxy is set as migrated 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] found 4 new records for migration_status 10:04 ERROR [databricks.labs.blueprint.parallel] migrate views(ViewToMigrate(src=Table(catalog='hive_metastore', database='ucx_spbzh', name='view3', object_type='VIEW', table_format='UNKNOWN', location=None, view_text='SELECT * FROM hive_metastore.ucx_spbzh.ucx_tcvxy', upgraded_to=None, storage_properties='{serialization.format: 1}', is_partitioned=False), rule=Rule(workspace_name='workspace', catalog_name='ucx_cqm4y', src_schema='ucx_spbzh', dst_schema='ucx_spbzh', src_table='view3', dst_table='view3', recon_tolerance_percent=0, compare_rows=False)), []) task failed: This transaction attempted to read one or more files that were deleted (for example part-00000-92d6a658-e942-4039-9896-2158d727b0ad.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720001076648,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1723","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"4","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"4","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"751999cb-021e-489c-81cb-926db318d79d"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 200, in _migrate_view if self._view_can_be_migrated(src_view): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 208, in _view_can_be_migrated if not self.index_full_refresh().get(table.schema, table.name): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 65, in index_full_refresh self._migration_status_refresher.reset() File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 55, in reset self._exec(f"DELETE FROM {self.full_name}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 142, in execute self._sql.execute(sql, catalog=catalog, schema=schema) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 265, in execute self._raise_if_needed(result_status) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 479, in _raise_if_needed raise error_class(error_message) databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92d6a658-e942-4039-9896-2158d727b0ad.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720001076648,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1723","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"4","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"4","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"751999cb-021e-489c-81cb-926db318d79d"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] crawling new batch for migration_status 10:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.tables] fetching tables inventory 10:05 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spbzh.ucx_tcvxy is set as migrated 10:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] found 4 new records for migration_status 10:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_swevs.migration_status] fetching migration_status inventory 10:06 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating view hive_metastore.ucx_spbzh.ucx_txhx9 to using SQL query: CREATE VIEW IF NOT EXISTS ucx_cqm4y.ucx_spbzh.ucx_txhx9 (id, value) TBLPROPERTIES ('RemoveAfter'='2024070311', 'transient_lastDdlTime'='1720000471') AS SELECT * FROM ucx_cqm4y.ucx_spbzh.ucx_tcvxy 10:06 ERROR [databricks.labs.blueprint.parallel] More than half 'migrate views' tasks failed: 50% results available (1/2). Took 0:02:26.078733 10:06 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 10:06 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000575130, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cqm4y', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cqm4y', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000575130, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:06 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:06 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:06 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 10:06 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_tcvxy', metastore_id=None, name='ucx_tcvxy', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_spbzh/ucx_tcvxy', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:07 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_txhx9', metastore_id=None, name='ucx_txhx9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_tcvxy', view_dependencies=None) 10:07 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_txhx9', metastore_id=None, name='ucx_txhx9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_tcvxy', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.ucx_spbzh.ucx_txhx9 is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:07 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_t1suw', metastore_id=None, name='ucx_t1suw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_txhx9', view_dependencies=None) 10:07 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spbzh.ucx_t1suw', metastore_id=None, name='ucx_t1suw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='ucx_spbzh', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spbzh.ucx_txhx9', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.ucx_spbzh.ucx_t1suw is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:07 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 10:07 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spbzh', metastore_id=None, name='ucx_spbzh', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cqm4y', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cqm4y.ucx_spbzh', metastore_id=None, name='ucx_spbzh', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swevs', metastore_id=None, name='ucx_swevs', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_table_migration_job_refreshes_migration_status[regular-migrate-tables]: TimeoutError: Timed out after 0:05:00 (17m13.353s) ``` TimeoutError: Timed out after 0:05:00 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qos5q: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_trnwy: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_trnwy 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_trnwy', metastore_id=None, name='ucx_trnwy', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qos5q/ucx_trnwy', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_t6tla: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_t6tla 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t6tla', metastore_id=None, name='ucx_t6tla', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_9Izq', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_t8ocm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_t8ocm 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t8ocm', metastore_id=None, name='ucx_t8ocm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:57 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_tonzz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_tonzz 09:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tonzz', metastore_id=None, name='ucx_tonzz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_trnwy', view_dependencies=None) 09:58 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_tiu0u: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_tiu0u 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tiu0u', metastore_id=None, name='ucx_tiu0u', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_tonzz', view_dependencies=None) 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000700555, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cpkt1', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000700555, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:58 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cpkt1.migrate_qos5q: https://DATABRICKS_HOST/explore/data/ucx_cpkt1/migrate_qos5q 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cpkt1', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:59 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgatt: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgatt 09:59 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgatt', metastore_id=None, name='ucx_sgatt', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qos5q: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_trnwy: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_trnwy 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_trnwy', metastore_id=None, name='ucx_trnwy', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qos5q/ucx_trnwy', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_t6tla: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_t6tla 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t6tla', metastore_id=None, name='ucx_t6tla', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_9Izq', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_t8ocm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_t8ocm 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t8ocm', metastore_id=None, name='ucx_t8ocm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:57 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_tonzz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_tonzz 09:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tonzz', metastore_id=None, name='ucx_tonzz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_trnwy', view_dependencies=None) 09:58 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_tiu0u: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_tiu0u 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tiu0u', metastore_id=None, name='ucx_tiu0u', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_tonzz', view_dependencies=None) 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000700555, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cpkt1', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000700555, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:58 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cpkt1.migrate_qos5q: https://DATABRICKS_HOST/explore/data/ucx_cpkt1/migrate_qos5q 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cpkt1', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:59 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgatt: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgatt 09:59 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgatt', metastore_id=None, name='ucx_sgatt', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:01 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.iJBn/config.yml) doesn't exist. 10:01 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:01 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:01 INFO [databricks.labs.ucx.install] Fetching installations... 10:01 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.iJBn is corrupted. Skipping... 10:01 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:01 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:01 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:01 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7220240703100118 10:01 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:04 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.iJBn/README for the next steps. 10:04 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/48642638122068 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qos5q: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_trnwy: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_trnwy 09:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_trnwy', metastore_id=None, name='ucx_trnwy', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qos5q/ucx_trnwy', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_t6tla: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_t6tla 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t6tla', metastore_id=None, name='ucx_t6tla', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_9Izq', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_t8ocm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_t8ocm 09:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t8ocm', metastore_id=None, name='ucx_t8ocm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:57 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_tonzz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_tonzz 09:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tonzz', metastore_id=None, name='ucx_tonzz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_trnwy', view_dependencies=None) 09:58 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qos5q.ucx_tiu0u: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qos5q/ucx_tiu0u 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tiu0u', metastore_id=None, name='ucx_tiu0u', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_tonzz', view_dependencies=None) 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000700555, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cpkt1', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000700555, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:58 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cpkt1.migrate_qos5q: https://DATABRICKS_HOST/explore/data/ucx_cpkt1/migrate_qos5q 09:58 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cpkt1', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:59 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgatt: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgatt 09:59 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgatt', metastore_id=None, name='ucx_sgatt', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:01 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.iJBn/config.yml) doesn't exist. 10:01 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:01 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:01 INFO [databricks.labs.ucx.install] Fetching installations... 10:01 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.iJBn is corrupted. Skipping... 10:01 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:01 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:01 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:01 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7220240703100118 10:01 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:04 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.iJBn/README for the next steps. 10:04 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/48642638122068 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 10:08 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000700555, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cpkt1', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000700555, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:08 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7220240703100118 from https://DATABRICKS_HOST 10:08 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sgatt 10:09 INFO [databricks.labs.ucx.install] Deleting jobs 10:09 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=389675670023711. 10:09 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=29722115191964. 10:09 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=48642638122068. 10:09 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=806692385702618. 10:09 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=31453556639316. 10:09 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=492169051400396. 10:09 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=336404130732212. 10:09 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=232295345852616. 10:09 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=931801528925798. 10:09 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=922567589056657. 10:09 INFO [databricks.labs.ucx.install] Deleting failing job_id=55762425232810. 10:09 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=477417121598862. 10:09 INFO [databricks.labs.ucx.install] Deleting assessment job_id=1030842462280673. 10:09 INFO [databricks.labs.ucx.install] Deleting cluster policy 10:09 INFO [databricks.labs.ucx.install] Deleting secret scope 10:09 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_trnwy', metastore_id=None, name='ucx_trnwy', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qos5q/ucx_trnwy', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t6tla', metastore_id=None, name='ucx_t6tla', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_9Izq', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:09 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_t8ocm', metastore_id=None, name='ucx_t8ocm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Sc3e', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:10 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tonzz', metastore_id=None, name='ucx_tonzz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_trnwy', view_dependencies=None) 10:10 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tonzz', metastore_id=None, name='ucx_tonzz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_trnwy', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_qos5q.ucx_tonzz is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:10 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tiu0u', metastore_id=None, name='ucx_tiu0u', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_tonzz', view_dependencies=None) 10:11 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qos5q.ucx_tiu0u', metastore_id=None, name='ucx_tiu0u', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_qos5q', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qos5q.ucx_tonzz', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_qos5q.ucx_tiu0u is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 10:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cpkt1', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkt1.migrate_qos5q', metastore_id=None, name='migrate_qos5q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgatt', metastore_id=None, name='ucx_sgatt', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_running_real_workflow_linter_job: TimeoutError: timed out after 0:20:00: (25m8.803s) ``` TimeoutError: timed out after 0:20:00: [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC/notebook.ipynb 09:52 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/974969198730 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=974969198730) 09:52 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svqmd: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svqmd 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svqmd', metastore_id=None, name='ucx_svqmd', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:52 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.b91I/config.yml) doesn't exist. 09:52 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:52 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:52 INFO [databricks.labs.ucx.install] Fetching installations... 09:52 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:52 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:52 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:52 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7220240703095246 09:52 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:56 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.b91I/README for the next steps. 09:56 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/63472138867105 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC/notebook.ipynb 09:52 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/974969198730 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=974969198730) 09:52 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svqmd: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svqmd 09:52 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svqmd', metastore_id=None, name='ucx_svqmd', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:52 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.b91I/config.yml) doesn't exist. 09:52 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:52 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:52 INFO [databricks.labs.ucx.install] Fetching installations... 09:52 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:52 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:52 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:52 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7220240703095246 09:52 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:56 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:56 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.b91I/README for the next steps. 09:56 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/63472138867105 10:16 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 job fixtures 10:16 DEBUG [databricks.labs.ucx.mixins.fixtures] removing job fixture: CreateResponse(job_id=974969198730) 10:16 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 directory fixtures 10:16 DEBUG [databricks.labs.ucx.mixins.fixtures] removing directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC 10:16 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 notebook fixtures 10:16 DEBUG [databricks.labs.ucx.mixins.fixtures] removing notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC/notebook.ipynb 10:16 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while notebook /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC/notebook.ipynb teardown: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-mZGC/notebook.ipynb) doesn't exist. 10:16 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7220240703095246 from https://DATABRICKS_HOST 10:16 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_svqmd 10:17 INFO [databricks.labs.ucx.install] Deleting jobs 10:17 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=63472138867105. 10:17 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=520636767448377. 10:17 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=241152632657491. 10:17 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=224515102847272. 10:17 INFO [databricks.labs.ucx.install] Deleting assessment job_id=95453457324654. 10:17 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=237845297166309. 10:17 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=519373704478673. 10:17 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=857344189361880. 10:17 INFO [databricks.labs.ucx.install] Deleting failing job_id=172153502615848. 10:17 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=882998555117815. 10:17 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=149186048495404. 10:17 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=1005982169624048. 10:17 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=1105327494365297. 10:17 INFO [databricks.labs.ucx.install] Deleting cluster policy 10:17 INFO [databricks.labs.ucx.install] Deleting secret scope 10:17 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 10:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 10:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 10:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 10:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svqmd', metastore_id=None, name='ucx_svqmd', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

Running from acceptance #4246