whitelist livereload - Githubissues

❌ 151/155 passed, 4 failed, 23 skipped, 6h4m2s total

❌ test_migrate_view: databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92a22097-f73f-46c6-9d41-86c39c074907.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. (10m44.159s)

``` databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92a22097-f73f-46c6-9d41-86c39c074907.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720513540216,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1722","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"11","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"3","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"445999af-3329-4eaa-9ac9-84e39a84e20e"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. [gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 08:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_spv3l: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l 08:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spv3l', metastore_id=None, name='ucx_spv3l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:18 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spv3l.ucx_txcii: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l/ucx_txcii 08:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_txcii', metastore_id=None, name='ucx_txcii', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_spv3l/ucx_txcii', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:18 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spv3l.ucx_t0qm4: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l/ucx_t0qm4 08:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_t0qm4', metastore_id=None, name='ucx_t0qm4', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_txcii', view_dependencies=None) 08:19 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spv3l.ucx_tqbl9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l/ucx_tqbl9 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_tqbl9', metastore_id=None, name='ucx_tqbl9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_t0qm4', view_dependencies=None) 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513158010, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkid', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cpkid', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513158010, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:19 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cpkid.ucx_spv3l: https://DATABRICKS_HOST/explore/data/ucx_cpkid/ucx_spv3l 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cpkid', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkid.ucx_spv3l', metastore_id=None, name='ucx_spv3l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:19 INFO [tests.integration.hive_metastore.test_migrate] dst_catalog=ucx_cpkid, managed_table=hive_metastore.ucx_spv3l.ucx_txcii 08:19 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s3f58: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s3f58 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3f58', metastore_id=None, name='ucx_s3f58', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:19 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] crawling new batch for migration_status 08:19 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] crawling new batch for tables 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l] listing tables 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.ucx_t0qm4] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.ucx_tqbl9] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.ucx_txcii] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.view3] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] found 4 new records for tables 08:20 WARNING [databricks.labs.ucx.hive_metastore.migration_status] Catalog ucx_crjt5 no longer exists. Skipping checking its migration status. 08:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] found 4 new records for migration_status 08:21 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:22 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating managed table hive_metastore.ucx_spv3l.ucx_txcii to ucx_cpkid.ucx_spv3l.ucx_txcii using SQL query: CREATE TABLE IF NOT EXISTS ucx_cpkid.ucx_spv3l.ucx_txcii DEEP CLONE hive_metastore.ucx_spv3l.ucx_txcii; 08:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:23 INFO [databricks.labs.ucx.hive_metastore.mapping] The intended target for hive_metastore.ucx_spv3l.ucx_txcii, ucx_cpkid.ucx_spv3l.ucx_txcii, already exists. 08:24 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:24 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] crawling new batch for migration_status 08:24 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:25 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spv3l.ucx_txcii is set as migrated 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] found 4 new records for migration_status 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:25 ERROR [databricks.labs.blueprint.parallel] migrate views(ViewToMigrate(src=Table(catalog='hive_metastore', database='ucx_spv3l', name='ucx_t0qm4', object_type='VIEW', table_format='UNKNOWN', location=None, view_text='SELECT * FROM hive_metastore.ucx_spv3l.ucx_txcii', upgraded_to=None, storage_properties='{serialization.format: 1}', is_partitioned=False), rule=Rule(workspace_name='workspace', catalog_name='ucx_cpkid', src_schema='ucx_spv3l', dst_schema='ucx_spv3l', src_table='ucx_t0qm4', dst_table='ucx_t0qm4', recon_tolerance_percent=0, compare_rows=False)), []) task failed: This transaction attempted to read one or more files that were deleted (for example part-00000-92a22097-f73f-46c6-9d41-86c39c074907.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720513540216,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1722","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"11","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"3","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"445999af-3329-4eaa-9ac9-84e39a84e20e"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 200, in _migrate_view if self._view_can_be_migrated(src_view): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 208, in _view_can_be_migrated if not self.index_full_refresh().get(table.schema, table.name): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 65, in index_full_refresh self._migration_status_refresher.reset() File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 55, in reset self._exec(f"DELETE FROM {self.full_name}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 197, in execute self._sql.execute(sql, catalog=catalog, schema=schema) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 243, in execute self._raise_if_needed(status) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 480, in _raise_if_needed raise error_class(error_message) databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92a22097-f73f-46c6-9d41-86c39c074907.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720513540216,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1722","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"11","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"3","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"445999af-3329-4eaa-9ac9-84e39a84e20e"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] crawling new batch for migration_status 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:26 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spv3l.ucx_txcii is set as migrated 08:26 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] found 4 new records for migration_status 08:27 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:27 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating view hive_metastore.ucx_spv3l.view3 to using SQL query: CREATE VIEW IF NOT EXISTS ucx_cpkid.ucx_spv3l.view3 (col1, col2) TBLPROPERTIES ('transient_lastDdlTime'='1720513147') AS SELECT * FROM ucx_cpkid.ucx_spv3l.ucx_txcii 08:27 ERROR [databricks.labs.blueprint.parallel] More than half 'migrate views' tasks failed: 50% results available (1/2). Took 0:02:25.129953 08:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_spv3l: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l 08:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spv3l', metastore_id=None, name='ucx_spv3l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:18 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spv3l.ucx_txcii: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l/ucx_txcii 08:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_txcii', metastore_id=None, name='ucx_txcii', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_spv3l/ucx_txcii', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:18 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spv3l.ucx_t0qm4: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l/ucx_t0qm4 08:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_t0qm4', metastore_id=None, name='ucx_t0qm4', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_txcii', view_dependencies=None) 08:19 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_spv3l.ucx_tqbl9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_spv3l/ucx_tqbl9 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_tqbl9', metastore_id=None, name='ucx_tqbl9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_t0qm4', view_dependencies=None) 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513158010, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkid', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cpkid', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513158010, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:19 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cpkid.ucx_spv3l: https://DATABRICKS_HOST/explore/data/ucx_cpkid/ucx_spv3l 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cpkid', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkid.ucx_spv3l', metastore_id=None, name='ucx_spv3l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:19 INFO [tests.integration.hive_metastore.test_migrate] dst_catalog=ucx_cpkid, managed_table=hive_metastore.ucx_spv3l.ucx_txcii 08:19 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s3f58: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s3f58 08:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3f58', metastore_id=None, name='ucx_s3f58', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext. 08:19 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:19 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] crawling new batch for migration_status 08:19 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] crawling new batch for tables 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l] listing tables 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.ucx_t0qm4] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.ucx_tqbl9] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.ucx_txcii] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_spv3l.view3] fetching table metadata 08:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] found 4 new records for tables 08:20 WARNING [databricks.labs.ucx.hive_metastore.migration_status] Catalog ucx_crjt5 no longer exists. Skipping checking its migration status. 08:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] found 4 new records for migration_status 08:21 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:22 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating managed table hive_metastore.ucx_spv3l.ucx_txcii to ucx_cpkid.ucx_spv3l.ucx_txcii using SQL query: CREATE TABLE IF NOT EXISTS ucx_cpkid.ucx_spv3l.ucx_txcii DEEP CLONE hive_metastore.ucx_spv3l.ucx_txcii; 08:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:23 INFO [databricks.labs.ucx.hive_metastore.mapping] The intended target for hive_metastore.ucx_spv3l.ucx_txcii, ucx_cpkid.ucx_spv3l.ucx_txcii, already exists. 08:24 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:24 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] crawling new batch for migration_status 08:24 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:25 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spv3l.ucx_txcii is set as migrated 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] found 4 new records for migration_status 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:25 ERROR [databricks.labs.blueprint.parallel] migrate views(ViewToMigrate(src=Table(catalog='hive_metastore', database='ucx_spv3l', name='ucx_t0qm4', object_type='VIEW', table_format='UNKNOWN', location=None, view_text='SELECT * FROM hive_metastore.ucx_spv3l.ucx_txcii', upgraded_to=None, storage_properties='{serialization.format: 1}', is_partitioned=False), rule=Rule(workspace_name='workspace', catalog_name='ucx_cpkid', src_schema='ucx_spv3l', dst_schema='ucx_spv3l', src_table='ucx_t0qm4', dst_table='ucx_t0qm4', recon_tolerance_percent=0, compare_rows=False)), []) task failed: This transaction attempted to read one or more files that were deleted (for example part-00000-92a22097-f73f-46c6-9d41-86c39c074907.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720513540216,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1722","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"11","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"3","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"445999af-3329-4eaa-9ac9-84e39a84e20e"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 200, in _migrate_view if self._view_can_be_migrated(src_view): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 208, in _view_can_be_migrated if not self.index_full_refresh().get(table.schema, table.name): File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py", line 65, in index_full_refresh self._migration_status_refresher.reset() File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 55, in reset self._exec(f"DELETE FROM {self.full_name}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 197, in execute self._sql.execute(sql, catalog=catalog, schema=schema) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 243, in execute self._raise_if_needed(status) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 480, in _raise_if_needed raise error_class(error_message) databricks.sdk.errors.platform.BadRequest: This transaction attempted to read one or more files that were deleted (for example part-00000-92a22097-f73f-46c6-9d41-86c39c074907.c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1720513540216,"userId":"481119220561874","userName":"0a330eb5-dd51-4d97-b6e4-c474356b1d5d","operation":"DELETE","operationParameters":{"predicate":["true"]},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numRemovedBytes":"1722","numCopiedRows":"0","numDeletionVectorsAdded":"0","numDeletionVectorsRemoved":"0","numAddedChangeFiles":"0","executionTimeMs":"11","numDeletionVectorsUpdated":"0","numDeletedRows":"4","scanTimeMs":"3","numAddedFiles":"0","numAddedBytes":"0","rewriteTimeMs":"0"},"tags":{"noRowsCopied":"true","delta.rowTracking.preserved":"false","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/15.1.x-photon-scala2.12","txnId":"445999af-3329-4eaa-9ac9-84e39a84e20e"} Refer to https://docs.microsoft.com/CLOUD_ENV/databricks/delta/concurrency-control for more details. 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] crawling new batch for migration_status 08:25 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.tables] fetching tables inventory 08:26 INFO [databricks.labs.ucx.hive_metastore.migration_status] ucx_spv3l.ucx_txcii is set as migrated 08:26 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] found 4 new records for migration_status 08:27 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3f58.migration_status] fetching migration_status inventory 08:27 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating view hive_metastore.ucx_spv3l.view3 to using SQL query: CREATE VIEW IF NOT EXISTS ucx_cpkid.ucx_spv3l.view3 (col1, col2) TBLPROPERTIES ('transient_lastDdlTime'='1720513147') AS SELECT * FROM ucx_cpkid.ucx_spv3l.ucx_txcii 08:27 ERROR [databricks.labs.blueprint.parallel] More than half 'migrate views' tasks failed: 50% results available (1/2). Took 0:02:25.129953 08:27 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 08:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513158010, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkid', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cpkid', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513158010, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:27 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 08:27 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 08:27 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures 08:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_txcii', metastore_id=None, name='ucx_txcii', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_spv3l/ucx_txcii', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_t0qm4', metastore_id=None, name='ucx_t0qm4', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_txcii', view_dependencies=None) 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_t0qm4', metastore_id=None, name='ucx_t0qm4', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_txcii', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.ucx_spv3l.ucx_t0qm4 is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_tqbl9', metastore_id=None, name='ucx_tqbl9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_t0qm4', view_dependencies=None) 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_spv3l.ucx_tqbl9', metastore_id=None, name='ucx_tqbl9', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='ucx_spv3l', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.ucx_spv3l.ucx_t0qm4', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.ucx_spv3l.ucx_tqbl9 is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_spv3l', metastore_id=None, name='ucx_spv3l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cpkid', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cpkid.ucx_spv3l', metastore_id=None, name='ucx_spv3l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3f58', metastore_id=None, name='ucx_s3f58', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

❌ test_table_migration_job_refreshes_migration_status[regular-migrate-tables]: AssertionError: No destination schema found for TableType.VIEW hive_metastore.migrate_vdi8t.ucx_t9xtd given migration statuses Row(src_schema='migrate_vdi8t', src_table='ucx_tvhij', dst_catalog='ucx_ckjp2', dst_schema='migrate_vdi8t', dst_table='ucx_tvhij', update_ts='1720514083.802202') (22m57.572s)

``` AssertionError: No destination schema found for TableType.VIEW hive_metastore.migrate_vdi8t.ucx_t9xtd given migration statuses Row(src_schema='migrate_vdi8t', src_table='ucx_tvhij', dst_catalog='ucx_ckjp2', dst_schema='migrate_vdi8t', dst_table='ucx_tvhij', update_ts='1720514083.802202') Row(src_schema='migrate_vdi8t', src_table='ucx_tooha', dst_catalog='ucx_ckjp2', dst_schema='migrate_vdi8t', dst_table='ucx_tooha', update_ts='1720514083.802202') Row(src_schema='migrate_vdi8t', src_table='ucx_tefgw', dst_catalog='ucx_ckjp2', dst_schema='migrate_vdi8t', dst_table='ucx_tefgw', update_ts='1720514083.802202') Row(src_schema='migrate_vdi8t', src_table='ucx_tfxsk', dst_catalog='ucx_ckjp2', dst_schema='migrate_vdi8t', dst_table='ucx_tfxsk', update_ts='1720514083.802202') Row(src_schema='migrate_vdi8t', src_table='ucx_t9xtd', dst_catalog=None, dst_schema=None, dst_table=None, update_ts='1720514083.802202') assert 1 == 0 + where 1 = len(['No destination schema found for TableType.VIEW hive_metastore.migrate_vdi8t.ucx_t9xtd']) 08:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_vdi8t: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tvhij: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tvhij 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tvhij', metastore_id=None, name='ucx_tvhij', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_vdi8t/ucx_tvhij', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tooha: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tooha 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tooha', metastore_id=None, name='ucx_tooha', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_B3P8', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tefgw: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tefgw 08:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tefgw', metastore_id=None, name='ucx_tefgw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tfxsk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tfxsk 08:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tfxsk', metastore_id=None, name='ucx_tfxsk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tvhij', view_dependencies=None) 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_t9xtd: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_t9xtd 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_t9xtd', metastore_id=None, name='ucx_t9xtd', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tfxsk', view_dependencies=None) 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513402953, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckjp2', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513402953, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_ckjp2.migrate_vdi8t: https://DATABRICKS_HOST/explore/data/ucx_ckjp2/migrate_vdi8t 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckjp2', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s5cxk: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s5cxk 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s5cxk', metastore_id=None, name='ucx_s5cxk', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 08:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_vdi8t: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tvhij: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tvhij 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tvhij', metastore_id=None, name='ucx_tvhij', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_vdi8t/ucx_tvhij', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tooha: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tooha 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tooha', metastore_id=None, name='ucx_tooha', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_B3P8', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tefgw: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tefgw 08:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tefgw', metastore_id=None, name='ucx_tefgw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tfxsk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tfxsk 08:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tfxsk', metastore_id=None, name='ucx_tfxsk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tvhij', view_dependencies=None) 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_t9xtd: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_t9xtd 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_t9xtd', metastore_id=None, name='ucx_t9xtd', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tfxsk', view_dependencies=None) 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513402953, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckjp2', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513402953, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_ckjp2.migrate_vdi8t: https://DATABRICKS_HOST/explore/data/ucx_ckjp2/migrate_vdi8t 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckjp2', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s5cxk: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s5cxk 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s5cxk', metastore_id=None, name='ucx_s5cxk', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:25 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.d2gs/config.yml) doesn't exist. 08:25 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 08:26 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 08:26 INFO [databricks.labs.ucx.install] Fetching installations... 08:26 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.d2gs is corrupted. Skipping... 08:26 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 08:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:26 INFO [databricks.labs.ucx.install] Installing UCX v0.28.1+2920240709082627 08:26 INFO [databricks.labs.ucx.install] Creating ucx schemas... 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 08:28 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.d2gs/README for the next steps. 08:28 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/532630402911378 08:20 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_vdi8t: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tvhij: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tvhij 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tvhij', metastore_id=None, name='ucx_tvhij', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_vdi8t/ucx_tvhij', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tooha: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tooha 08:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tooha', metastore_id=None, name='ucx_tooha', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_B3P8', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tefgw: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tefgw 08:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tefgw', metastore_id=None, name='ucx_tefgw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:22 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_tfxsk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_tfxsk 08:22 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tfxsk', metastore_id=None, name='ucx_tfxsk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tvhij', view_dependencies=None) 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_vdi8t.ucx_t9xtd: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_vdi8t/ucx_t9xtd 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_t9xtd', metastore_id=None, name='ucx_t9xtd', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tfxsk', view_dependencies=None) 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513402953, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckjp2', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513402953, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_ckjp2.migrate_vdi8t: https://DATABRICKS_HOST/explore/data/ucx_ckjp2/migrate_vdi8t 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckjp2', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:23 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s5cxk: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s5cxk 08:23 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s5cxk', metastore_id=None, name='ucx_s5cxk', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:25 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.d2gs/config.yml) doesn't exist. 08:25 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 08:26 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 08:26 INFO [databricks.labs.ucx.install] Fetching installations... 08:26 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.d2gs is corrupted. Skipping... 08:26 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 08:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:26 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:26 INFO [databricks.labs.ucx.install] Installing UCX v0.28.1+2920240709082627 08:26 INFO [databricks.labs.ucx.install] Creating ucx schemas... 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 08:28 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 08:28 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.d2gs/README for the next steps. 08:28 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/532630402911378 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513402953, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_ckjp2', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513402953, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:42 INFO [databricks.labs.ucx.install] Deleting UCX v0.28.1+2920240709082627 from https://DATABRICKS_HOST 08:42 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s5cxk 08:42 INFO [databricks.labs.ucx.install] Deleting jobs 08:42 INFO [databricks.labs.ucx.install] Deleting assessment job_id=454033111182728. 08:42 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=745284370387559. 08:42 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=623553930758311. 08:42 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=1050913427923799. 08:42 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=895860029049048. 08:42 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=164511193646191. 08:42 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=712501886853010. 08:42 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=532630402911378. 08:42 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=360813102297681. 08:42 INFO [databricks.labs.ucx.install] Deleting failing job_id=145904903100814. 08:42 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=1002220728056909. 08:42 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=616191116305742. 08:42 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=64439521478738. 08:42 INFO [databricks.labs.ucx.install] Deleting cluster policy 08:42 INFO [databricks.labs.ucx.install] Deleting secret scope 08:42 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tvhij', metastore_id=None, name='ucx_tvhij', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_vdi8t/ucx_tvhij', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tooha', metastore_id=None, name='ucx_tooha', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_B3P8', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tefgw', metastore_id=None, name='ucx_tefgw', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/QMsY', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tfxsk', metastore_id=None, name='ucx_tfxsk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tvhij', view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_tfxsk', metastore_id=None, name='ucx_tfxsk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tvhij', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_vdi8t.ucx_tfxsk is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_t9xtd', metastore_id=None, name='ucx_t9xtd', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tfxsk', view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_vdi8t.ucx_t9xtd', metastore_id=None, name='ucx_t9xtd', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_vdi8t', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_vdi8t.ucx_tfxsk', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_vdi8t.ucx_t9xtd is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_ckjp2', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_ckjp2.migrate_vdi8t', metastore_id=None, name='migrate_vdi8t', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s5cxk', metastore_id=None, name='ucx_s5cxk', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw2] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

❌ test_table_migration_job_cluster_override[regular]: AssertionError: assert False (14m59.057s)

``` AssertionError: assert False + where False = >('migrate-tables') + where > = .validate_step + where = .deployed_workflows 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI 08:29 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_sulfv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:29 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tog4w: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tog4w 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tog4w', metastore_id=None, name='ucx_tog4w', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_sulfv/ucx_tog4w', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_t2gwm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_t2gwm 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_t2gwm', metastore_id=None, name='ucx_t2gwm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_ddG0', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_td0gr: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_td0gr 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_td0gr', metastore_id=None, name='ucx_td0gr', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tqfom: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tqfom 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tqfom', metastore_id=None, name='ucx_tqfom', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tog4w', view_dependencies=None) 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tfcga: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tfcga 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tfcga', metastore_id=None, name='ucx_tfcga', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tqfom', view_dependencies=None) 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513877512, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwcse', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513877512, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cwcse.migrate_sulfv: https://DATABRICKS_HOST/explore/data/ucx_cwcse/migrate_sulfv 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwcse', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_suet6: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_suet6 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suet6', metastore_id=None, name='ucx_suet6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw5] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI 08:29 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_sulfv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:29 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tog4w: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tog4w 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tog4w', metastore_id=None, name='ucx_tog4w', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_sulfv/ucx_tog4w', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_t2gwm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_t2gwm 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_t2gwm', metastore_id=None, name='ucx_t2gwm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_ddG0', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_td0gr: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_td0gr 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_td0gr', metastore_id=None, name='ucx_td0gr', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tqfom: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tqfom 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tqfom', metastore_id=None, name='ucx_tqfom', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tog4w', view_dependencies=None) 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tfcga: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tfcga 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tfcga', metastore_id=None, name='ucx_tfcga', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tqfom', view_dependencies=None) 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513877512, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwcse', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513877512, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cwcse.migrate_sulfv: https://DATABRICKS_HOST/explore/data/ucx_cwcse/migrate_sulfv 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwcse', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_suet6: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_suet6 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suet6', metastore_id=None, name='ucx_suet6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:33 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.kbNd/config.yml) doesn't exist. 08:33 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 08:33 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 08:33 INFO [databricks.labs.ucx.install] Fetching installations... 08:33 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.kbNd is corrupted. Skipping... 08:33 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 08:33 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:33 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:33 INFO [databricks.labs.ucx.install] Installing UCX v0.28.1+2920240709083307 08:33 INFO [databricks.labs.ucx.install] Creating ucx schemas... 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 08:35 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.kbNd/README for the next steps. 08:35 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/592025548195572 08:43 DEBUG [databricks.labs.ucx.installer.workflows] Validating migrate-tables workflow: https://DATABRICKS_HOST#job/592025548195572 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI 08:29 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_sulfv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:29 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tog4w: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tog4w 08:29 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tog4w', metastore_id=None, name='ucx_tog4w', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_sulfv/ucx_tog4w', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_t2gwm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_t2gwm 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_t2gwm', metastore_id=None, name='ucx_t2gwm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_ddG0', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_td0gr: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_td0gr 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_td0gr', metastore_id=None, name='ucx_td0gr', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:30 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tqfom: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tqfom 08:30 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tqfom', metastore_id=None, name='ucx_tqfom', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tog4w', view_dependencies=None) 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_sulfv.ucx_tfcga: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_sulfv/ucx_tfcga 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tfcga', metastore_id=None, name='ucx_tfcga', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tqfom', view_dependencies=None) 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513877512, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwcse', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513877512, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cwcse.migrate_sulfv: https://DATABRICKS_HOST/explore/data/ucx_cwcse/migrate_sulfv 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwcse', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:31 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_suet6: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_suet6 08:31 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suet6', metastore_id=None, name='ucx_suet6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:33 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.kbNd/config.yml) doesn't exist. 08:33 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 08:33 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 08:33 INFO [databricks.labs.ucx.install] Fetching installations... 08:33 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.kbNd is corrupted. Skipping... 08:33 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 08:33 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:33 DEBUG [tests.integration.conftest] Waiting for clusters to start... 08:33 INFO [databricks.labs.ucx.install] Installing UCX v0.28.1+2920240709083307 08:33 INFO [databricks.labs.ucx.install] Creating ucx schemas... 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 08:35 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 08:35 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.kbNd/README for the next steps. 08:35 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/592025548195572 08:43 DEBUG [databricks.labs.ucx.installer.workflows] Validating migrate-tables workflow: https://DATABRICKS_HOST#job/592025548195572 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=, comment='', connection_name=None, created_at=1720513877512, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse', isolation_mode=, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwcse', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1720513877512, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') 08:43 INFO [databricks.labs.ucx.install] Deleting UCX v0.28.1+2920240709083307 from https://DATABRICKS_HOST 08:43 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_suet6 08:43 INFO [databricks.labs.ucx.install] Deleting jobs 08:43 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=177327929147126. 08:43 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=335698399799327. 08:43 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=429976909532268. 08:43 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=282235348948743. 08:43 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=592025548195572. 08:43 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=38273774455487. 08:43 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=1021679588268533. 08:43 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=324437793958004. 08:43 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=11069635334057. 08:43 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=698958839988944. 08:43 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=720448056963562. 08:43 INFO [databricks.labs.ucx.install] Deleting failing job_id=1085690257689124. 08:43 INFO [databricks.labs.ucx.install] Deleting assessment job_id=1086177258156438. 08:43 INFO [databricks.labs.ucx.install] Deleting cluster policy 08:43 INFO [databricks.labs.ucx.install] Deleting secret scope 08:43 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tog4w', metastore_id=None, name='ucx_tog4w', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_sulfv/ucx_tog4w', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_t2gwm', metastore_id=None, name='ucx_t2gwm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_ddG0', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_td0gr', metastore_id=None, name='ucx_td0gr', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ckfI', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tqfom', metastore_id=None, name='ucx_tqfom', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tog4w', view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tqfom', metastore_id=None, name='ucx_tqfom', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tog4w', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_sulfv.ucx_tqfom is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tfcga', metastore_id=None, name='ucx_tfcga', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tqfom', view_dependencies=None) 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_sulfv.ucx_tfcga', metastore_id=None, name='ucx_tfcga', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070909'}, row_filter=None, schema_name='migrate_sulfv', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_sulfv.ucx_tqfom', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_sulfv.ucx_tfcga is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures 08:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwcse', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwcse.migrate_sulfv', metastore_id=None, name='migrate_sulfv', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 08:44 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_suet6', metastore_id=None, name='ucx_suet6', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw5] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

❌ test_models_root_page[False]: TimeoutError: Timed out after 0:05:00 (5m4.253s)

``` TimeoutError: Timed out after 0:05:00 08:47 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-FMQE: https://DATABRICKS_HOST#setting/accounts/groups/430195451106482 08:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-FMQE', entitlements=[], external_id=None, groups=[], id='430195451106482', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 08:47 INFO [databricks.labs.ucx.mixins.fixtures] Account group sdk-eEud: https://accounts.CLOUD_ENVdatabricks.net/users/groups/908182842165494/members 08:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='sdk-eEud', entitlements=[], external_id=None, groups=[], id='908182842165494', members=[], meta=None, roles=[], schemas=[]) 08:47 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-FMQE: https://DATABRICKS_HOST#setting/accounts/groups/430195451106482 08:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-FMQE', entitlements=[], external_id=None, groups=[], id='430195451106482', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 08:47 INFO [databricks.labs.ucx.mixins.fixtures] Account group sdk-eEud: https://accounts.CLOUD_ENVdatabricks.net/users/groups/908182842165494/members 08:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='sdk-eEud', entitlements=[], external_id=None, groups=[], id='908182842165494', members=[], meta=None, roles=[], schemas=[]) 08:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures 08:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='sdk-eEud', entitlements=[], external_id=None, groups=[], id='908182842165494', members=[], meta=None, roles=[], schemas=[]) 08:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 08:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='sdk-FMQE', entitlements=[], external_id=None, groups=[], id='430195451106482', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) [gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

_{Running from acceptance #4427}

databrickslabs / ucx

whitelist livereload #2052

Changes

Linked issues

Functionality

Tests