search

databrickslabs / ucx

Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Other
194 stars 69 forks source link

whitelist rpy2 #2033

Closed ericvergnaud closed 2 days ago

ericvergnaud commented 4 days ago

Changes

whitelist rpy2

Linked issues

Progresses #1901

Functionality

None

Tests

github-actions[bot] commented 4 days ago

❌ 90/93 passed, 3 failed, 21 skipped, 5h7m9s total

❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: TimeoutError: Timed out after 0:05:00 (16m5.956s) ``` TimeoutError: Timed out after 0:05:00 10:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/c7DM 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.parquet_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/parquet_serde_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.parquet_serde_dmdto', metastore_id=None, name='parquet_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/parquet_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.orc_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/orc_serde_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.orc_serde_dmdto', metastore_id=None, name='orc_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/orc_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.avro_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/avro_serde_dmdto 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.avro_serde_dmdto', metastore_id=None, name='avro_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/avro_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720002132306, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cajdp', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720002132306, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cajdp.hiveserde_in_place_dmdto: https://DATABRICKS_HOST/explore/data/ucx_cajdp/hiveserde_in_place_dmdto 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cajdp', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sj9es: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sj9es 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sj9es', metastore_id=None, name='ucx_sj9es', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 10:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/c7DM 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.parquet_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/parquet_serde_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.parquet_serde_dmdto', metastore_id=None, name='parquet_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/parquet_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.orc_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/orc_serde_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.orc_serde_dmdto', metastore_id=None, name='orc_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/orc_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.avro_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/avro_serde_dmdto 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.avro_serde_dmdto', metastore_id=None, name='avro_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/avro_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720002132306, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cajdp', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720002132306, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cajdp.hiveserde_in_place_dmdto: https://DATABRICKS_HOST/explore/data/ucx_cajdp/hiveserde_in_place_dmdto 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cajdp', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sj9es: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sj9es 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sj9es', metastore_id=None, name='ucx_sj9es', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:25 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.24QK/config.yml) doesn't exist. 10:25 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:25 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:25 INFO [databricks.labs.ucx.install] Fetching installations... 10:25 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.24QK is corrupted. Skipping... 10:25 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:25 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:25 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:25 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7020240703102542 10:25 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:28 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.24QK/README for the next steps. 10:28 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/856591532992782 10:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/c7DM 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.parquet_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/parquet_serde_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.parquet_serde_dmdto', metastore_id=None, name='parquet_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/parquet_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.orc_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/orc_serde_dmdto 10:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.orc_serde_dmdto', metastore_id=None, name='orc_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/orc_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_dmdto.avro_serde_dmdto: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_dmdto/avro_serde_dmdto 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.avro_serde_dmdto', metastore_id=None, name='avro_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/avro_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720002132306, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cajdp', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720002132306, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cajdp.hiveserde_in_place_dmdto: https://DATABRICKS_HOST/explore/data/ucx_cajdp/hiveserde_in_place_dmdto 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cajdp', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:22 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sj9es: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sj9es 10:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sj9es', metastore_id=None, name='ucx_sj9es', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:25 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.24QK/config.yml) doesn't exist. 10:25 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:25 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:25 INFO [databricks.labs.ucx.install] Fetching installations... 10:25 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.24QK is corrupted. Skipping... 10:25 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:25 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:25 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:25 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7020240703102542 10:25 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:28 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.24QK/README for the next steps. 10:28 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/856591532992782 10:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_storage_dir fixtures 10:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto 10:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 10:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/c7DM 10:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 10:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 10:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720002132306, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cajdp', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720002132306, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7020240703102542 from https://DATABRICKS_HOST 10:34 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sj9es 10:35 INFO [databricks.labs.ucx.install] Deleting jobs 10:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=679737773253568. 10:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=1102917999677559. 10:35 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=420870765095836. 10:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=165098972795058. 10:35 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=969782274094804. 10:35 INFO [databricks.labs.ucx.install] Deleting failing job_id=231441243489048. 10:35 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=644086993023064. 10:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=856591532992782. 10:35 INFO [databricks.labs.ucx.install] Deleting assessment job_id=527960469778729. 10:35 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=134815754929521. 10:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=300705458879705. 10:35 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=830654605286930. 10:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=384213556932955. 10:35 INFO [databricks.labs.ucx.install] Deleting cluster policy 10:35 INFO [databricks.labs.ucx.install] Deleting secret scope 10:35 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.parquet_serde_dmdto', metastore_id=None, name='parquet_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/parquet_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.orc_serde_dmdto', metastore_id=None, name='orc_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/orc_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_dmdto.avro_serde_dmdto', metastore_id=None, name='avro_serde_dmdto', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='hiveserde_in_place_dmdto', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_dmdto/avro_serde_dmdto', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cajdp', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cajdp.hiveserde_in_place_dmdto', metastore_id=None, name='hiveserde_in_place_dmdto', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sj9es', metastore_id=None, name='ucx_sj9es', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_running_real_workflow_linter_job: TimeoutError: Timed out after 0:02:00 (9m9.265s) ``` TimeoutError: Timed out after 0:02:00 [gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO/notebook.ipynb 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/345497819608887 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=345497819608887) 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_snlie: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_snlie 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_snlie', metastore_id=None, name='ucx_snlie', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:35 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.MTDS/config.yml) doesn't exist. 10:35 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:35 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:35 INFO [databricks.labs.ucx.install] Fetching installations... 10:35 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:36 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:36 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:36 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7020240703103602 10:36 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:39 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.MTDS/README for the next steps. 10:39 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/271656288272430 10:43 DEBUG [databricks.labs.ucx.installer.workflows] Validating experimental-workflow-linter workflow: https://DATABRICKS_HOST#job/271656288272430 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO/notebook.ipynb 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/345497819608887 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=345497819608887) 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_snlie: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_snlie 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_snlie', metastore_id=None, name='ucx_snlie', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:35 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.MTDS/config.yml) doesn't exist. 10:35 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:35 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:35 INFO [databricks.labs.ucx.install] Fetching installations... 10:35 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:36 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:36 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:36 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7020240703103602 10:36 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:38 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:39 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.MTDS/README for the next steps. 10:39 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/271656288272430 10:43 DEBUG [databricks.labs.ucx.installer.workflows] Validating experimental-workflow-linter workflow: https://DATABRICKS_HOST#job/271656288272430 10:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 job fixtures 10:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing job fixture: CreateResponse(job_id=345497819608887) 10:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 directory fixtures 10:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO 10:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 notebook fixtures 10:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO/notebook.ipynb 10:43 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while notebook /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO/notebook.ipynb teardown: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-ERcO/notebook.ipynb) doesn't exist. 10:43 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7020240703103602 from https://DATABRICKS_HOST 10:43 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_snlie 10:44 INFO [databricks.labs.ucx.install] Deleting jobs 10:44 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=259466588314845. 10:44 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=271656288272430. 10:44 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=183330544496677. 10:44 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=158407563040051. 10:44 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=570108514834682. 10:44 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=392462026000685. 10:44 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=874039702744708. 10:44 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=874383975045529. 10:44 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=46587712214255. 10:44 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=420245653555630. 10:44 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=603357764657870. 10:44 INFO [databricks.labs.ucx.install] Deleting assessment job_id=730410388880627. 10:44 INFO [databricks.labs.ucx.install] Deleting failing job_id=330466732993980. 10:44 INFO [databricks.labs.ucx.install] Deleting cluster policy 10:44 INFO [databricks.labs.ucx.install] Deleting secret scope 10:44 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 10:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 10:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 10:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:44 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 10:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_snlie', metastore_id=None, name='ucx_snlie', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_table_migration_job_refreshes_migration_status[regular-migrate-tables]: TimeoutError: Timed out after 0:05:00 (22m32.57s) ``` TimeoutError: Timed out after 0:05:00 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_g7ewm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tchjv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tchjv 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tchjv', metastore_id=None, name='ucx_tchjv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_g7ewm/ucx_tchjv', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tvijv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tvijv 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvijv', metastore_id=None, name='ucx_tvijv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_D0eE', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tu493: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tu493 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tu493', metastore_id=None, name='ucx_tu493', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:37 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tvcf0: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tvcf0 10:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvcf0', metastore_id=None, name='ucx_tvcf0', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tchjv', view_dependencies=None) 10:38 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_txtcz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_txtcz 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_txtcz', metastore_id=None, name='ucx_txtcz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tvcf0', view_dependencies=None) 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720003103015, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckqip', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720003103015, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_ckqip.migrate_g7ewm: https://DATABRICKS_HOST/explore/data/ucx_ckqip/migrate_g7ewm 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckqip', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:39 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_suqh7: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_suqh7 10:39 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suqh7', metastore_id=None, name='ucx_suqh7', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_g7ewm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tchjv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tchjv 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tchjv', metastore_id=None, name='ucx_tchjv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_g7ewm/ucx_tchjv', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tvijv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tvijv 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvijv', metastore_id=None, name='ucx_tvijv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_D0eE', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tu493: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tu493 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tu493', metastore_id=None, name='ucx_tu493', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:37 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tvcf0: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tvcf0 10:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvcf0', metastore_id=None, name='ucx_tvcf0', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tchjv', view_dependencies=None) 10:38 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_txtcz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_txtcz 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_txtcz', metastore_id=None, name='ucx_txtcz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tvcf0', view_dependencies=None) 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720003103015, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckqip', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720003103015, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_ckqip.migrate_g7ewm: https://DATABRICKS_HOST/explore/data/ucx_ckqip/migrate_g7ewm 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckqip', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:39 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_suqh7: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_suqh7 10:39 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suqh7', metastore_id=None, name='ucx_suqh7', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:41 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.qwDF/config.yml) doesn't exist. 10:41 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:41 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:41 INFO [databricks.labs.ucx.install] Fetching installations... 10:42 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.qwDF is corrupted. Skipping... 10:42 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:42 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:42 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:42 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7020240703104210 10:42 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:46 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.qwDF/README for the next steps. 10:46 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/197388006248211 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_g7ewm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:35 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tchjv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tchjv 10:35 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tchjv', metastore_id=None, name='ucx_tchjv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_g7ewm/ucx_tchjv', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tvijv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tvijv 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvijv', metastore_id=None, name='ucx_tvijv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_D0eE', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:36 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tu493: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tu493 10:36 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tu493', metastore_id=None, name='ucx_tu493', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:37 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_tvcf0: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_tvcf0 10:37 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvcf0', metastore_id=None, name='ucx_tvcf0', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tchjv', view_dependencies=None) 10:38 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_g7ewm.ucx_txtcz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_g7ewm/ucx_txtcz 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_txtcz', metastore_id=None, name='ucx_txtcz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tvcf0', view_dependencies=None) 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720003103015, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckqip', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720003103015, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_ckqip.migrate_g7ewm: https://DATABRICKS_HOST/explore/data/ucx_ckqip/migrate_g7ewm 10:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckqip', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:39 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_suqh7: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_suqh7 10:39 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suqh7', metastore_id=None, name='ucx_suqh7', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:41 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.qwDF/config.yml) doesn't exist. 10:41 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 10:41 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 10:41 INFO [databricks.labs.ucx.install] Fetching installations... 10:42 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.qwDF is corrupted. Skipping... 10:42 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 10:42 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:42 DEBUG [tests.integration.conftest] Waiting for clusters to start... 10:42 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7020240703104210 10:42 INFO [databricks.labs.ucx.install] Creating ucx schemas... 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 10:46 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 10:46 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.qwDF/README for the next steps. 10:46 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/197388006248211 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720003103015, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckqip', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720003103015, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:54 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7020240703104210 from https://DATABRICKS_HOST 10:54 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_suqh7 10:54 INFO [databricks.labs.ucx.install] Deleting jobs 10:54 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=866504546099842. 10:54 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=698375827184564. 10:54 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=279715517263618. 10:54 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=589970836798175. 10:54 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=609473642260865. 10:54 INFO [databricks.labs.ucx.install] Deleting failing job_id=648924780860544. 10:54 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=218675014898604. 10:54 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=1038293486275794. 10:54 INFO [databricks.labs.ucx.install] Deleting assessment job_id=709620020162411. 10:54 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=221272103746071. 10:54 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=937714059048117. 10:54 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=197388006248211. 10:54 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=364909485847942. 10:54 INFO [databricks.labs.ucx.install] Deleting cluster policy 10:54 INFO [databricks.labs.ucx.install] Deleting secret scope 10:54 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures 10:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tchjv', metastore_id=None, name='ucx_tchjv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_g7ewm/ucx_tchjv', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvijv', metastore_id=None, name='ucx_tvijv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_D0eE', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tu493', metastore_id=None, name='ucx_tu493', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/0xGn', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvcf0', metastore_id=None, name='ucx_tvcf0', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tchjv', view_dependencies=None) 10:56 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_tvcf0', metastore_id=None, name='ucx_tvcf0', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tchjv', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_g7ewm.ucx_tvcf0 is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:56 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_txtcz', metastore_id=None, name='ucx_txtcz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tvcf0', view_dependencies=None) 10:56 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_g7ewm.ucx_txtcz', metastore_id=None, name='ucx_txtcz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_g7ewm', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_g7ewm.ucx_tvcf0', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_g7ewm.ucx_txtcz is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:56 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 10:56 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:57 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckqip', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckqip.migrate_g7ewm', metastore_id=None, name='migrate_g7ewm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:57 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suqh7', metastore_id=None, name='ucx_suqh7', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

Running from acceptance #4254