search

databrickslabs / ucx

Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Other
193 stars 69 forks source link

whitelist altair #2005

Closed ericvergnaud closed 2 days ago

ericvergnaud commented 2 days ago

Changes

whitelist altair

Linked issues

Progresses #1901

Functionality

None

Tests

github-actions[bot] commented 2 days ago

❌ 67/69 passed, 2 failed, 17 skipped, 4h21m17s total

❌ test_all_grant_types: databricks.sdk.errors.platform.BadRequest: [SCHEMA_ALREADY_EXISTS] Cannot create schema `ucx_s8r0c` because it already exists. (5m47.256s) ``` databricks.sdk.errors.platform.BadRequest: [SCHEMA_ALREADY_EXISTS] Cannot create schema `ucx_s8r0c` because it already exists. Choose a different name, drop the existing schema, or add the IF NOT EXISTS clause to tolerate pre-existing schema. SQLSTATE: 42P06 16:28 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s8ubu: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s8ubu 16:28 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s8ubu', metastore_id=None, name='ucx_s8ubu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:28 INFO [databricks.labs.ucx.install] Creating ucx schemas... [gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 16:28 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s8ubu: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s8ubu 16:28 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s8ubu', metastore_id=None, name='ucx_s8ubu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:28 INFO [databricks.labs.ucx.install] Creating ucx schemas... 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-O1jj: https://DATABRICKS_HOST#setting/accounts/groups/447345938964598 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-O1jj', entitlements=[], external_id=None, groups=[], id='447345938964598', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 16:28 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s8ubu: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s8ubu 16:28 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s8ubu', metastore_id=None, name='ucx_s8ubu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:28 INFO [databricks.labs.ucx.install] Creating ucx schemas... 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-O1jj: https://DATABRICKS_HOST#setting/accounts/groups/447345938964598 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-O1jj', entitlements=[], external_id=None, groups=[], id='447345938964598', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='sdk-O1jj', entitlements=[], external_id=None, groups=[], id='447345938964598', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s8ubu', metastore_id=None, name='ucx_s8ubu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: TimeoutError: Timed out after 0:05:00 (18m2.534s) ``` TimeoutError: Timed out after 0:05:00 16:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/9g0i 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.parquet_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/parquet_serde_dyxtu 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.parquet_serde_dyxtu', metastore_id=None, name='parquet_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/parquet_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:34 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.orc_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/orc_serde_dyxtu 16:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.orc_serde_dyxtu', metastore_id=None, name='orc_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/orc_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:34 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.avro_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/avro_serde_dyxtu 16:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.avro_serde_dyxtu', metastore_id=None, name='avro_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/avro_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719938106892, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cjs83', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938106892, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 16:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cjs83.hiveserde_in_place_dyxtu: https://DATABRICKS_HOST/explore/data/ucx_cjs83/hiveserde_in_place_dyxtu 16:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cjs83', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:36 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sm5th: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sm5th 16:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sm5th', metastore_id=None, name='ucx_sm5th', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 16:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/9g0i 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.parquet_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/parquet_serde_dyxtu 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.parquet_serde_dyxtu', metastore_id=None, name='parquet_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/parquet_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:34 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.orc_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/orc_serde_dyxtu 16:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.orc_serde_dyxtu', metastore_id=None, name='orc_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/orc_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:34 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.avro_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/avro_serde_dyxtu 16:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.avro_serde_dyxtu', metastore_id=None, name='avro_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/avro_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719938106892, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cjs83', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938106892, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 16:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cjs83.hiveserde_in_place_dyxtu: https://DATABRICKS_HOST/explore/data/ucx_cjs83/hiveserde_in_place_dyxtu 16:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cjs83', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:36 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sm5th: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sm5th 16:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sm5th', metastore_id=None, name='ucx_sm5th', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:39 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.f4hB/config.yml) doesn't exist. 16:39 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 16:39 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 16:39 INFO [databricks.labs.ucx.install] Fetching installations... 16:39 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.f4hB is corrupted. Skipping... 16:39 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 16:39 DEBUG [tests.integration.conftest] Waiting for clusters to start... 16:39 DEBUG [tests.integration.conftest] Waiting for clusters to start... 16:39 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+4220240702163953 16:39 INFO [databricks.labs.ucx.install] Creating ucx schemas... 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 16:43 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.f4hB/README for the next steps. 16:43 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/90619917269195 16:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/9g0i 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu 16:33 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.parquet_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/parquet_serde_dyxtu 16:33 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.parquet_serde_dyxtu', metastore_id=None, name='parquet_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/parquet_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:34 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.orc_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/orc_serde_dyxtu 16:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.orc_serde_dyxtu', metastore_id=None, name='orc_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/orc_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:34 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dyxtu.avro_serde_dyxtu: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dyxtu/avro_serde_dyxtu 16:34 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.avro_serde_dyxtu', metastore_id=None, name='avro_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/avro_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719938106892, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cjs83', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938106892, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 16:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cjs83.hiveserde_in_place_dyxtu: https://DATABRICKS_HOST/explore/data/ucx_cjs83/hiveserde_in_place_dyxtu 16:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cjs83', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:36 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sm5th: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sm5th 16:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sm5th', metastore_id=None, name='ucx_sm5th', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:39 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.f4hB/config.yml) doesn't exist. 16:39 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 16:39 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 16:39 INFO [databricks.labs.ucx.install] Fetching installations... 16:39 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.f4hB is corrupted. Skipping... 16:39 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 16:39 DEBUG [tests.integration.conftest] Waiting for clusters to start... 16:39 DEBUG [tests.integration.conftest] Waiting for clusters to start... 16:39 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+4220240702163953 16:39 INFO [databricks.labs.ucx.install] Creating ucx schemas... 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 16:43 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 16:43 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.f4hB/README for the next steps. 16:43 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/90619917269195 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_storage_dir fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/9g0i 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719938106892, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cjs83', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938106892, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 16:48 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+4220240702163953 from https://DATABRICKS_HOST 16:48 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sm5th 16:48 INFO [databricks.labs.ucx.install] Deleting jobs 16:48 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=882145144501265. 16:48 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=866586560363141. 16:48 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=522459169754897. 16:48 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=90619917269195. 16:48 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=231120091715784. 16:48 INFO [databricks.labs.ucx.install] Deleting failing job_id=661368034150401. 16:48 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=31402715440429. 16:48 INFO [databricks.labs.ucx.install] Deleting assessment job_id=985118558940993. 16:48 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=318227378005357. 16:48 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=769982776936385. 16:48 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=782560375995513. 16:48 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=1063946612419428. 16:48 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=687201119637633. 16:48 INFO [databricks.labs.ucx.install] Deleting cluster policy 16:48 INFO [databricks.labs.ucx.install] Deleting secret scope 16:48 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.parquet_serde_dyxtu', metastore_id=None, name='parquet_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/parquet_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.orc_serde_dyxtu', metastore_id=None, name='orc_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/orc_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dyxtu.avro_serde_dyxtu', metastore_id=None, name='avro_serde_dyxtu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070217'}, row_filter=None, schema_name='hiveserde_in_place_dyxtu', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dyxtu/avro_serde_dyxtu', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cjs83', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cjs83.hiveserde_in_place_dyxtu', metastore_id=None, name='hiveserde_in_place_dyxtu', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sm5th', metastore_id=None, name='ucx_sm5th', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

Running from acceptance #4174