whitelist dbldatagen - Githubissues

❌ 83/86 passed, 3 failed, 18 skipped, 5h45m35s total

❌ test_spn_crawler: databricks.sdk.errors.platform.BadRequest: [SCHEMA_ALREADY_EXISTS] Cannot create schema `ucx_scny6` because it already exists. (25.679s)

``` databricks.sdk.errors.platform.BadRequest: [SCHEMA_ALREADY_EXISTS] Cannot create schema `ucx_scny6` because it already exists. Choose a different name, drop the existing schema, or add the IF NOT EXISTS clause to tolerate pre-existing schema. SQLSTATE: 42P06 15:32 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 schema fixtures [gw0] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

❌ test_create_catalog_schema_with_principal_acl_CLOUD_ENV: databricks.sdk.errors.platform.BadRequest: [RequestId=32ecd6d5-2ee1-4d3e-92b5-522a994b276e ErrorClass=PRINCIPAL_DOES_NOT_EXIST] Could not find principal with name ucx_GCtj5 (14m36.462s)

``` databricks.sdk.errors.platform.BadRequest: [RequestId=32ecd6d5-2ee1-4d3e-92b5-522a994b276e ErrorClass=PRINCIPAL_DOES_NOT_EXIST] Could not find principal with name ucx_GCtj5 15:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf 15:33 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sobdy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sobdy 15:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:33 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sobdy.ucx_tazcf: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sobdy/ucx_tazcf 15:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sobdy.ucx_tazcf', metastore_id=None, name='ucx_tazcf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='ucx_sobdy', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719934449525, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cycaa', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719934449525, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:34 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cycaa.ucx_sobdy: https://DATABRICKS_HOST/explore/data/ucx_cycaa/ucx_sobdy 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cycaa', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 15:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf 15:33 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sobdy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sobdy 15:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:33 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sobdy.ucx_tazcf: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sobdy/ucx_tazcf 15:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sobdy.ucx_tazcf', metastore_id=None, name='ucx_tazcf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='ucx_sobdy', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719934449525, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cycaa', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719934449525, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:34 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cycaa.ucx_sobdy: https://DATABRICKS_HOST/explore/data/ucx_cycaa/ucx_sobdy 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cycaa', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-jegukook@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jegukook@example.com')], entitlements=[], external_id=None, groups=[], id='5302423103526659', name=Name(family_name=None, given_name='sdk-jegukook@example.com'), roles=[], schemas=[, ], user_name='sdk-jegukook@example.com') 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-153438-3kbzoyys [group_name admins CAN_MANAGE] -> [user_name sdk-jegukook@example.com CAN_ATTACH_TO] 15:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sk6pj: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sk6pj 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sk6pj', metastore_id=None, name='ucx_sk6pj', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas 15:35 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Preparing a list of UC catalogs and schema to be created 15:35 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sk6pj.tables] fetching tables inventory 15:36 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sk6pj.tables] crawling new batch for tables 15:36 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sobdy] listing tables 15:37 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sobdy.ucx_tazcf] fetching table metadata 15:37 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sk6pj.tables] found 1 new records for tables 15:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_GLpCH` 15:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-jegukook@example.com` 15:40 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-okk9sifo@example.com` 15:40 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_Gs40O` 15:41 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-ftdkz7ws@example.com` 15:41 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `serge.smertin@databricks.com` 15:41 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-1nzgua2i@example.com` 15:42 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-qvqdsgxo@example.com` 15:42 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_GcM0H` 15:42 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d` 15:43 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `5a11359f-ba1f-483f-8e00-0fe55ec003ed` 15:43 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-rvilnco2@example.com` 15:44 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_GCtj5` 15:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf 15:33 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sobdy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sobdy 15:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:33 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sobdy.ucx_tazcf: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sobdy/ucx_tazcf 15:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sobdy.ucx_tazcf', metastore_id=None, name='ucx_tazcf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='ucx_sobdy', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719934449525, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cycaa', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719934449525, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:34 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cycaa.ucx_sobdy: https://DATABRICKS_HOST/explore/data/ucx_cycaa/ucx_sobdy 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cycaa', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-jegukook@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jegukook@example.com')], entitlements=[], external_id=None, groups=[], id='5302423103526659', name=Name(family_name=None, given_name='sdk-jegukook@example.com'), roles=[], schemas=[, ], user_name='sdk-jegukook@example.com') 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-153438-3kbzoyys [group_name admins CAN_MANAGE] -> [user_name sdk-jegukook@example.com CAN_ATTACH_TO] 15:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sk6pj: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sk6pj 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sk6pj', metastore_id=None, name='ucx_sk6pj', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 15:35 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas 15:35 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Preparing a list of UC catalogs and schema to be created 15:35 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sk6pj.tables] fetching tables inventory 15:36 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sk6pj.tables] crawling new batch for tables 15:36 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sobdy] listing tables 15:37 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sobdy.ucx_tazcf] fetching table metadata 15:37 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sk6pj.tables] found 1 new records for tables 15:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_GLpCH` 15:39 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-jegukook@example.com` 15:40 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-okk9sifo@example.com` 15:40 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_Gs40O` 15:41 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-ftdkz7ws@example.com` 15:41 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `serge.smertin@databricks.com` 15:41 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-1nzgua2i@example.com` 15:42 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-qvqdsgxo@example.com` 15:42 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_GcM0H` 15:42 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d` 15:43 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `5a11359f-ba1f-483f-8e00-0fe55ec003ed` 15:43 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `sdk-rvilnco2@example.com` 15:44 DEBUG [databricks.labs.ucx.hive_metastore.catalog_schema] Migrating acls on ('DATABASE', 'ucx_cycaa.ucx_sobdy') using SQL query: GRANT USE SCHEMA ON DATABASE ucx_cycaa.ucx_sobdy TO `ucx_GCtj5` 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster permissions fixtures 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster permissions fixture: 0702-153438-3kbzoyys [group_name admins CAN_MANAGE] -> [user_name sdk-jegukook@example.com CAN_ATTACH_TO] 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719934449525, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cycaa', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719934449525, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster fixtures 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster fixture: 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 table fixtures 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sobdy.ucx_tazcf', metastore_id=None, name='ucx_tazcf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='ucx_sobdy', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/dLWf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:45 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 15:45 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:45 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cycaa', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cycaa.ucx_sobdy', metastore_id=None, name='ucx_sobdy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:46 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sk6pj', metastore_id=None, name='ucx_sk6pj', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:46 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures 15:46 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-jegukook@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jegukook@example.com')], entitlements=[], external_id=None, groups=[], id='5302423103526659', name=Name(family_name=None, given_name='sdk-jegukook@example.com'), roles=[], schemas=[, ], user_name='sdk-jegukook@example.com') [gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: TimeoutError: Timed out after 0:05:00 (19m47.454s)

``` TimeoutError: Timed out after 0:05:00 15:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/aaOH 15:42 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll 15:42 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.parquet_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/parquet_serde_gxcll 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.parquet_serde_gxcll', metastore_id=None, name='parquet_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/parquet_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.orc_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/orc_serde_gxcll 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.orc_serde_gxcll', metastore_id=None, name='orc_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/orc_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.avro_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/avro_serde_gxcll 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.avro_serde_gxcll', metastore_id=None, name='avro_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/avro_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719935031804, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cbyde', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719935031804, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cbyde.hiveserde_in_place_gxcll: https://DATABRICKS_HOST/explore/data/ucx_cbyde/hiveserde_in_place_gxcll 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cbyde', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_swbaw: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swbaw 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swbaw', metastore_id=None, name='ucx_swbaw', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 15:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/aaOH 15:42 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll 15:42 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.parquet_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/parquet_serde_gxcll 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.parquet_serde_gxcll', metastore_id=None, name='parquet_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/parquet_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.orc_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/orc_serde_gxcll 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.orc_serde_gxcll', metastore_id=None, name='orc_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/orc_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.avro_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/avro_serde_gxcll 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.avro_serde_gxcll', metastore_id=None, name='avro_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/avro_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719935031804, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cbyde', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719935031804, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cbyde.hiveserde_in_place_gxcll: https://DATABRICKS_HOST/explore/data/ucx_cbyde/hiveserde_in_place_gxcll 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cbyde', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_swbaw: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swbaw 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swbaw', metastore_id=None, name='ucx_swbaw', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:48 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.QxUx/config.yml) doesn't exist. 15:48 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 15:48 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 15:48 INFO [databricks.labs.ucx.install] Fetching installations... 15:48 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.QxUx is corrupted. Skipping... 15:49 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 15:49 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:49 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:49 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+2820240702154951 15:49 INFO [databricks.labs.ucx.install] Creating ucx schemas... 15:53 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 15:53 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 15:53 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 15:54 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.QxUx/README for the next steps. 15:54 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/511966254863947 15:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/aaOH 15:42 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll 15:42 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.parquet_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/parquet_serde_gxcll 15:42 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.parquet_serde_gxcll', metastore_id=None, name='parquet_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/parquet_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.orc_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/orc_serde_gxcll 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.orc_serde_gxcll', metastore_id=None, name='orc_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/orc_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_gxcll.avro_serde_gxcll: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_gxcll/avro_serde_gxcll 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.avro_serde_gxcll', metastore_id=None, name='avro_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/avro_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:43 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719935031804, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cbyde', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719935031804, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cbyde.hiveserde_in_place_gxcll: https://DATABRICKS_HOST/explore/data/ucx_cbyde/hiveserde_in_place_gxcll 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cbyde', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:44 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_swbaw: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swbaw 15:44 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swbaw', metastore_id=None, name='ucx_swbaw', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:48 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.QxUx/config.yml) doesn't exist. 15:48 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 15:48 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 15:48 INFO [databricks.labs.ucx.install] Fetching installations... 15:48 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.QxUx is corrupted. Skipping... 15:49 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 15:49 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:49 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:49 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+2820240702154951 15:49 INFO [databricks.labs.ucx.install] Creating ucx schemas... 15:53 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 15:53 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 15:53 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 15:54 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 15:54 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.QxUx/README for the next steps. 15:54 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/511966254863947 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_storage_dir fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/aaOH 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719935031804, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cbyde', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719935031804, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:58 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+2820240702154951 from https://DATABRICKS_HOST 15:58 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_swbaw 15:58 INFO [databricks.labs.ucx.install] Deleting jobs 15:58 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=512504927829845. 15:58 INFO [databricks.labs.ucx.install] Deleting assessment job_id=3993864477322. 15:58 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=215700307681362. 15:58 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=362064220285002. 15:58 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=648141155014343. 15:58 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=336955084967338. 15:58 INFO [databricks.labs.ucx.install] Deleting failing job_id=301888260016707. 15:58 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=437148878400907. 15:58 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=252769706284202. 15:58 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=40565970801097. 15:58 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=205493929192022. 15:58 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=511966254863947. 15:58 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=30995019318064. 15:58 INFO [databricks.labs.ucx.install] Deleting cluster policy 15:58 INFO [databricks.labs.ucx.install] Deleting secret scope 15:58 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 15:58 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.parquet_serde_gxcll', metastore_id=None, name='parquet_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/parquet_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:59 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.orc_serde_gxcll', metastore_id=None, name='orc_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/orc_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:59 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_gxcll.avro_serde_gxcll', metastore_id=None, name='avro_serde_gxcll', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_gxcll', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_gxcll/avro_serde_gxcll', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:00 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 16:00 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:00 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cbyde', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cbyde.hiveserde_in_place_gxcll', metastore_id=None, name='hiveserde_in_place_gxcll', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:01 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swbaw', metastore_id=None, name='ucx_swbaw', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

_{Running from acceptance #4140}

databrickslabs / ucx

whitelist dbldatagen #1985

Changes

Linked issues

Functionality

Tests