search

databrickslabs / ucx

Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Other
193 stars 69 forks source link

whitelist numba #1978

Closed ericvergnaud closed 2 days ago

ericvergnaud commented 2 days ago

Changes

whitelist numba

Linked issues

Progresses #1901

Functionality

None

Tests

github-actions[bot] commented 2 days ago

❌ 86/88 passed, 2 failed, 20 skipped, 4h42m31s total

❌ test_run_external_locations_using_access_connector: databricks.sdk.errors.platform.BadRequest: Storage Credential 'ac-labsCLOUD_ENVTEST_MOUNT_NAME' already exists (48.593s) ``` databricks.sdk.errors.platform.BadRequest: Storage Credential 'ac-labsCLOUD_ENVTEST_MOUNT_NAME' already exists [gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 15:08 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/QKB9 15:08 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sxypy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sxypy 15:08 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sxypy', metastore_id=None, name='ucx_sxypy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:08 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/QKB9 15:08 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sxypy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sxypy 15:08 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sxypy', metastore_id=None, name='ucx_sxypy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 15:09 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/QKB9 15:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 15:09 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 15:09 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sxypy', metastore_id=None, name='ucx_sxypy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: TimeoutError: Timed out after 0:05:00 (24m14.585s) ``` TimeoutError: Timed out after 0:05:00 15:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/euHa 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.parquet_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/parquet_serde_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.parquet_serde_bsph6', metastore_id=None, name='parquet_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/parquet_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.orc_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/orc_serde_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.orc_serde_bsph6', metastore_id=None, name='orc_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/orc_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.avro_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/avro_serde_bsph6 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.avro_serde_bsph6', metastore_id=None, name='avro_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/avro_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719933382672, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdwcd', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719933382672, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cdwcd.hiveserde_in_place_bsph6: https://DATABRICKS_HOST/explore/data/ucx_cdwcd/hiveserde_in_place_bsph6 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdwcd', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sueqn: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sueqn 15:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sueqn', metastore_id=None, name='ucx_sueqn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 15:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/euHa 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.parquet_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/parquet_serde_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.parquet_serde_bsph6', metastore_id=None, name='parquet_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/parquet_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.orc_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/orc_serde_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.orc_serde_bsph6', metastore_id=None, name='orc_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/orc_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.avro_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/avro_serde_bsph6 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.avro_serde_bsph6', metastore_id=None, name='avro_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/avro_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719933382672, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdwcd', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719933382672, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cdwcd.hiveserde_in_place_bsph6: https://DATABRICKS_HOST/explore/data/ucx_cdwcd/hiveserde_in_place_bsph6 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdwcd', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sueqn: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sueqn 15:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sueqn', metastore_id=None, name='ucx_sueqn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:20 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.M5Ok/config.yml) doesn't exist. 15:20 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 15:22 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 15:22 INFO [databricks.labs.ucx.install] Fetching installations... 15:22 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.M5Ok is corrupted. Skipping... 15:26 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 15:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:26 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+2820240702152601 15:26 INFO [databricks.labs.ucx.install] Creating ucx schemas... 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 15:30 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.M5Ok/README for the next steps. 15:30 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/1035873193797264 15:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/euHa 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.parquet_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/parquet_serde_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.parquet_serde_bsph6', metastore_id=None, name='parquet_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/parquet_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.orc_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/orc_serde_bsph6 15:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.orc_serde_bsph6', metastore_id=None, name='orc_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/orc_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bsph6.avro_serde_bsph6: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bsph6/avro_serde_bsph6 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.avro_serde_bsph6', metastore_id=None, name='avro_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/avro_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719933382672, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdwcd', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719933382672, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cdwcd.hiveserde_in_place_bsph6: https://DATABRICKS_HOST/explore/data/ucx_cdwcd/hiveserde_in_place_bsph6 15:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdwcd', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sueqn: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sueqn 15:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sueqn', metastore_id=None, name='ucx_sueqn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:20 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.M5Ok/config.yml) doesn't exist. 15:20 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 15:22 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 15:22 INFO [databricks.labs.ucx.install] Fetching installations... 15:22 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.M5Ok is corrupted. Skipping... 15:26 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 15:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 15:26 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+2820240702152601 15:26 INFO [databricks.labs.ucx.install] Creating ucx schemas... 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 15:30 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 15:30 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.M5Ok/README for the next steps. 15:30 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/1035873193797264 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_storage_dir fixtures 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/euHa 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 15:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1719933382672, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdwcd', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719933382672, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 15:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+2820240702152601 from https://DATABRICKS_HOST 15:34 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sueqn 15:35 INFO [databricks.labs.ucx.install] Deleting jobs 15:35 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=634693146219962. 15:35 INFO [databricks.labs.ucx.install] Deleting failing job_id=260291494111264. 15:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=770705549758747. 15:35 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=745762803579327. 15:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=487392277437215. 15:35 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=837486534324685. 15:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=1035873193797264. 15:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=1077381859069446. 15:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=358209485338072. 15:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=900377212158520. 15:35 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=690091628670162. 15:35 INFO [databricks.labs.ucx.install] Deleting assessment job_id=786002550783137. 15:35 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=630090501181874. 15:35 INFO [databricks.labs.ucx.install] Deleting cluster policy 15:35 INFO [databricks.labs.ucx.install] Deleting secret scope 15:35 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 15:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.parquet_serde_bsph6', metastore_id=None, name='parquet_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/parquet_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.orc_serde_bsph6', metastore_id=None, name='orc_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/orc_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bsph6.avro_serde_bsph6', metastore_id=None, name='avro_serde_bsph6', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070216'}, row_filter=None, schema_name='hiveserde_in_place_bsph6', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bsph6/avro_serde_bsph6', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 15:37 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 15:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:38 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdwcd', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdwcd.hiveserde_in_place_bsph6', metastore_id=None, name='hiveserde_in_place_bsph6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 15:38 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sueqn', metastore_id=None, name='ucx_sueqn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

Running from acceptance #4133