search

databrickslabs / ucx

Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Other
193 stars 69 forks source link

whitelist petastorm #1996

Closed ericvergnaud closed 17 hours ago

ericvergnaud commented 2 days ago

Changes

whitelist petastorm

Linked issues

Progresses #1901

Functionality

None

Tests

github-actions[bot] commented 2 days ago

❌ 103/106 passed, 3 failed, 22 skipped, 5h20m9s total

❌ test_running_real_workflow_linter_job: TimeoutError: Timed out after 0:02:00 (8m59.78s) ``` TimeoutError: Timed out after 0:02:00 [gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc/notebook.ipynb 09:45 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/803184953916854 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=803184953916854) 09:46 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svriv: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svriv 09:46 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svriv', metastore_id=None, name='ucx_svriv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:46 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Rx5w/config.yml) doesn't exist. 09:46 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:46 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:46 INFO [databricks.labs.ucx.install] Fetching installations... 09:46 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:46 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:46 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:46 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7120240703094639 09:46 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:49 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Rx5w/README for the next steps. 09:49 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/497641417420393 09:53 DEBUG [databricks.labs.ucx.installer.workflows] Validating experimental-workflow-linter workflow: https://DATABRICKS_HOST#job/497641417420393 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc/notebook.ipynb 09:45 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/803184953916854 09:45 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=803184953916854) 09:46 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svriv: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svriv 09:46 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svriv', metastore_id=None, name='ucx_svriv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:46 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Rx5w/config.yml) doesn't exist. 09:46 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:46 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:46 INFO [databricks.labs.ucx.install] Fetching installations... 09:46 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:46 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:46 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:46 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7120240703094639 09:46 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:49 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:49 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Rx5w/README for the next steps. 09:49 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/497641417420393 09:53 DEBUG [databricks.labs.ucx.installer.workflows] Validating experimental-workflow-linter workflow: https://DATABRICKS_HOST#job/497641417420393 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 job fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing job fixture: CreateResponse(job_id=803184953916854) 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 directory fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 notebook fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc/notebook.ipynb 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while notebook /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc/notebook.ipynb teardown: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-9rSc/notebook.ipynb) doesn't exist. 09:54 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7120240703094639 from https://DATABRICKS_HOST 09:54 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_svriv 09:54 INFO [databricks.labs.ucx.install] Deleting jobs 09:54 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=66118184465752. 09:54 INFO [databricks.labs.ucx.install] Deleting failing job_id=214727981680577. 09:54 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=895519745515543. 09:54 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=956651968378625. 09:54 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=788750729529696. 09:54 INFO [databricks.labs.ucx.install] Deleting assessment job_id=479837722112146. 09:54 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=908551190558138. 09:54 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=432735576107509. 09:54 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=497641417420393. 09:54 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=647372344427073. 09:54 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=996267413199008. 09:54 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=92856668320328. 09:54 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=218356787628428. 09:54 INFO [databricks.labs.ucx.install] Deleting cluster policy 09:54 INFO [databricks.labs.ucx.install] Deleting secret scope 09:54 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 09:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svriv', metastore_id=None, name='ucx_svriv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_migrate_table_in_mount: databricks.sdk.errors.platform.BadRequest: [SCHEMA_ALREADY_EXISTS] Cannot create schema `mounted_TEST_MOUNT_NAME` because it already exists. (31.238s) ``` databricks.sdk.errors.platform.BadRequest: [SCHEMA_ALREADY_EXISTS] Cannot create schema `mounted_TEST_MOUNT_NAME` because it already exists. Choose a different name, drop the existing schema, or add the IF NOT EXISTS clause to tolerate pre-existing schema. SQLSTATE: 42P06 10:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_splju: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_splju 10:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_splju', metastore_id=None, name='ucx_splju', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw0] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 10:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_splju: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_splju 10:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_splju', metastore_id=None, name='ucx_splju', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_splju: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_splju 10:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_splju', metastore_id=None, name='ucx_splju', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 catalog fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_splju', metastore_id=None, name='ucx_splju', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw0] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```
❌ test_table_migration_job_refreshes_migration_status[regular-migrate-tables]: TimeoutError: Timed out after 0:05:00 (18m17.512s) ``` TimeoutError: Timed out after 0:05:00 09:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs 09:48 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_7vhb4: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:48 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tqwnf: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tqwnf 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tqwnf', metastore_id=None, name='ucx_tqwnf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_7vhb4/ucx_tqwnf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:49 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tuueb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tuueb 09:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tuueb', metastore_id=None, name='ucx_tuueb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7u87', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:49 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_t4ekz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_t4ekz 09:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_t4ekz', metastore_id=None, name='ucx_t4ekz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:50 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tvier: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tvier 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tvier', metastore_id=None, name='ucx_tvier', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tqwnf', view_dependencies=None) 09:50 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tt3ey: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tt3ey 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tt3ey', metastore_id=None, name='ucx_tt3ey', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tvier', view_dependencies=None) 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000252123, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cxaug', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000252123, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:51 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cxaug.migrate_7vhb4: https://DATABRICKS_HOST/explore/data/ucx_cxaug/migrate_7vhb4 09:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cxaug', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:51 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sogej: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sogej 09:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sogej', metastore_id=None, name='ucx_sogej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 09:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs 09:48 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_7vhb4: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:48 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tqwnf: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tqwnf 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tqwnf', metastore_id=None, name='ucx_tqwnf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_7vhb4/ucx_tqwnf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:49 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tuueb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tuueb 09:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tuueb', metastore_id=None, name='ucx_tuueb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7u87', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:49 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_t4ekz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_t4ekz 09:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_t4ekz', metastore_id=None, name='ucx_t4ekz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:50 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tvier: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tvier 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tvier', metastore_id=None, name='ucx_tvier', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tqwnf', view_dependencies=None) 09:50 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tt3ey: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tt3ey 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tt3ey', metastore_id=None, name='ucx_tt3ey', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tvier', view_dependencies=None) 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000252123, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cxaug', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000252123, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:51 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cxaug.migrate_7vhb4: https://DATABRICKS_HOST/explore/data/ucx_cxaug/migrate_7vhb4 09:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cxaug', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:51 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sogej: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sogej 09:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sogej', metastore_id=None, name='ucx_sogej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:54 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XWxz/config.yml) doesn't exist. 09:54 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:54 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:54 INFO [databricks.labs.ucx.install] Fetching installations... 09:54 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XWxz is corrupted. Skipping... 09:54 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:54 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:54 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:54 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7120240703095429 09:54 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:57 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XWxz/README for the next steps. 09:57 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/109757106625102 09:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs 09:48 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_7vhb4: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:48 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tqwnf: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tqwnf 09:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tqwnf', metastore_id=None, name='ucx_tqwnf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_7vhb4/ucx_tqwnf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:49 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tuueb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tuueb 09:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tuueb', metastore_id=None, name='ucx_tuueb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7u87', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:49 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_t4ekz: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_t4ekz 09:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_t4ekz', metastore_id=None, name='ucx_t4ekz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 09:50 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tvier: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tvier 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tvier', metastore_id=None, name='ucx_tvier', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tqwnf', view_dependencies=None) 09:50 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_7vhb4.ucx_tt3ey: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_7vhb4/ucx_tt3ey 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tt3ey', metastore_id=None, name='ucx_tt3ey', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tvier', view_dependencies=None) 09:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000252123, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cxaug', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000252123, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 09:51 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cxaug.migrate_7vhb4: https://DATABRICKS_HOST/explore/data/ucx_cxaug/migrate_7vhb4 09:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cxaug', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:51 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sogej: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sogej 09:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sogej', metastore_id=None, name='ucx_sogej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 09:54 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XWxz/config.yml) doesn't exist. 09:54 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 09:54 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 09:54 INFO [databricks.labs.ucx.install] Fetching installations... 09:54 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XWxz is corrupted. Skipping... 09:54 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 09:54 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:54 DEBUG [tests.integration.conftest] Waiting for clusters to start... 09:54 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+7120240703095429 09:54 INFO [databricks.labs.ucx.install] Creating ucx schemas... 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 09:57 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 09:57 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XWxz/README for the next steps. 09:57 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/109757106625102 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720000252123, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cxaug', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720000252123, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 10:04 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+7120240703095429 from https://DATABRICKS_HOST 10:04 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sogej 10:04 INFO [databricks.labs.ucx.install] Deleting jobs 10:04 INFO [databricks.labs.ucx.install] Deleting assessment job_id=786503275286847. 10:04 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=786738127738829. 10:04 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=569360736783552. 10:04 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=449590287225225. 10:04 INFO [databricks.labs.ucx.install] Deleting failing job_id=73414679195921. 10:04 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=1036986550357507. 10:04 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=340526072382552. 10:04 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=799359726842615. 10:04 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=481530065255581. 10:04 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=307201529456904. 10:04 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=364679325350789. 10:04 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=684793662205511. 10:04 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=109757106625102. 10:04 INFO [databricks.labs.ucx.install] Deleting cluster policy 10:04 INFO [databricks.labs.ucx.install] Deleting secret scope 10:04 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tqwnf', metastore_id=None, name='ucx_tqwnf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_7vhb4/ucx_tqwnf', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tuueb', metastore_id=None, name='ucx_tuueb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_7u87', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:04 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_t4ekz', metastore_id=None, name='ucx_t4ekz', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/fpGs', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 10:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tvier', metastore_id=None, name='ucx_tvier', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tqwnf', view_dependencies=None) 10:05 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tvier', metastore_id=None, name='ucx_tvier', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tqwnf', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_7vhb4.ucx_tvier is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tt3ey', metastore_id=None, name='ucx_tt3ey', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tvier', view_dependencies=None) 10:05 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_7vhb4.ucx_tt3ey', metastore_id=None, name='ucx_tt3ey', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070311'}, row_filter=None, schema_name='migrate_7vhb4', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_7vhb4.ucx_tvier', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_7vhb4.ucx_tt3ey is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 10:05 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 10:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cxaug', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cxaug.migrate_7vhb4', metastore_id=None, name='migrate_7vhb4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 10:06 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sogej', metastore_id=None, name='ucx_sogej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

Running from acceptance #4244