Test failure: `test_running_real_assessment_job_ext_hms`

❌ test_running_real_assessment_job_ext_hms: databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ResourceDoesNotExist: Can't find a cluster policy with id: 001D9C7E67DB4BD9., TimeoutError: Timed out after 0:02:00 (15m57.797s)

``` databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ResourceDoesNotExist: Can't find a cluster policy with id: 001D9C7E67DB4BD9., TimeoutError: Timed out after 0:02:00 TimeoutError: Timed out after 0:05:00 [gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-txfe-ra78a52fd7@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-txfe-ra78a52fd7@example.com')], entitlements=[], external_id=None, groups=[], id='7321681649694012', name=Name(family_name=None, given_name='sdk-txfe-ra78a52fd7@example.com'), roles=[], schemas=[, ], user_name='sdk-txfe-ra78a52fd7@example.com') 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-lKCB-ra78a52fd7: https://DATABRICKS_HOST#setting/accounts/groups/86070713970517 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-lKCB-ra78a52fd7', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='86070713970517', members=[ComplexValue(display='sdk-txfe-ra78a52fd7@example.com', primary=None, ref='Users/7321681649694012', type=None, value='7321681649694012')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-lKCB-ra78a52fd7: https://accounts.CLOUD_ENVdatabricks.net/users/groups/772554636059199/members 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-lKCB-ra78a52fd7', entitlements=[], external_id=None, groups=[], id='772554636059199', members=[ComplexValue(display='sdk-txfe-ra78a52fd7@example.com', primary=None, ref='Users/7321681649694012', type=None, value='7321681649694012')], meta=None, roles=[], schemas=[]) 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000B797302611957 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000B797302611957') 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000B797302611957 [group_name admins CAN_USE] -> [group_name ucx-lKCB-ra78a52fd7 CAN_USE] 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_smuvn: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_smuvn 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smuvn', metastore_id=None, name='ucx_smuvn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 05:07 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.nErs/config.yml) doesn't exist. 05:07 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 05:07 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 05:07 INFO [databricks.labs.ucx.install] Fetching installations... 05:08 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore 05:08 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 05:08 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:15 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:15 INFO [databricks.labs.ucx.install] Installing UCX v0.36.1+420240916051514 05:15 INFO [databricks.labs.ucx.install] Creating ucx schemas... 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 05:15 ERROR [databricks.labs.blueprint.parallel] installing components task failed: Can't find a cluster policy with id: 001D9C7E67DB4BD9. Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/installer/workflows.py", line 479, in create_jobs self._deploy_workflow(workflow_name, settings) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/installer/workflows.py", line 595, in _deploy_workflow new_job = self._ws.jobs.create(**settings) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/jobs.py", line 5742, in create res = self._api.do('POST', '/api/2.1/jobs/create', body=body, headers=headers) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 155, in do response = retryable(self._perform)(method, File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 273, in _perform raise error from None databricks.sdk.errors.platform.ResourceDoesNotExist: Can't find a cluster policy with id: 001D9C7E67DB4BD9. 05:18 ERROR [databricks.labs.blueprint.parallel] installing components task failed: Timed out after 0:02:00 Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 540, in _create_database raise err File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 519, in _create_database deploy_schema(self._sql_backend, self._config.inventory_database) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 97, in deploy_schema deployer.deploy_schema() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/deployment.py", line 18, in deploy_schema self._sql_backend.execute(f"CREATE SCHEMA IF NOT EXISTS hive_metastore.{self._inventory_schema}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 221, in execute self._sql.run(sql) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/commands.py", line 113, in run ).result() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/_internal.py", line 67, in result return self._waiter(callback=callback, timeout=timeout, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/compute.py", line 7831, in wait_command_status_command_execution_finished_or_error poll = self.command_status(cluster_id=cluster_id, command_id=command_id, context_id=context_id) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/compute.py", line 7945, in command_status res = self._api.do('GET', '/api/1.2/commands/status', query=query, headers=headers) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 155, in do response = retryable(self._perform)(method, File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 273, in _perform raise error from None databricks.sdk.errors.platform.InternalError: TimeoutException: Timed out after 15 seconds The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 512, in _create_database_and_dashboards self._create_database() # Need the database before creating the dashboards File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 59, in wrapper raise TimeoutError(f'Timed out after {timeout}') from last_err TimeoutError: Timed out after 0:02:00 05:18 CRITICAL [databricks.labs.blueprint.parallel] All 'installing components' tasks failed!!! 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-txfe-ra78a52fd7@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-txfe-ra78a52fd7@example.com')], entitlements=[], external_id=None, groups=[], id='7321681649694012', name=Name(family_name=None, given_name='sdk-txfe-ra78a52fd7@example.com'), roles=[], schemas=[, ], user_name='sdk-txfe-ra78a52fd7@example.com') 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-lKCB-ra78a52fd7: https://DATABRICKS_HOST#setting/accounts/groups/86070713970517 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-lKCB-ra78a52fd7', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='86070713970517', members=[ComplexValue(display='sdk-txfe-ra78a52fd7@example.com', primary=None, ref='Users/7321681649694012', type=None, value='7321681649694012')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-lKCB-ra78a52fd7: https://accounts.CLOUD_ENVdatabricks.net/users/groups/772554636059199/members 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-lKCB-ra78a52fd7', entitlements=[], external_id=None, groups=[], id='772554636059199', members=[ComplexValue(display='sdk-txfe-ra78a52fd7@example.com', primary=None, ref='Users/7321681649694012', type=None, value='7321681649694012')], meta=None, roles=[], schemas=[]) 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000B797302611957 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000B797302611957') 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000B797302611957 [group_name admins CAN_USE] -> [group_name ucx-lKCB-ra78a52fd7 CAN_USE] 05:07 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_smuvn: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_smuvn 05:07 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smuvn', metastore_id=None, name='ucx_smuvn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 05:07 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.nErs/config.yml) doesn't exist. 05:07 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 05:07 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 05:07 INFO [databricks.labs.ucx.install] Fetching installations... 05:08 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore 05:08 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 05:08 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:15 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:15 INFO [databricks.labs.ucx.install] Installing UCX v0.36.1+420240916051514 05:15 INFO [databricks.labs.ucx.install] Creating ucx schemas... 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 05:15 ERROR [databricks.labs.blueprint.parallel] installing components task failed: Can't find a cluster policy with id: 001D9C7E67DB4BD9. Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/installer/workflows.py", line 479, in create_jobs self._deploy_workflow(workflow_name, settings) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/installer/workflows.py", line 595, in _deploy_workflow new_job = self._ws.jobs.create(**settings) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/jobs.py", line 5742, in create res = self._api.do('POST', '/api/2.1/jobs/create', body=body, headers=headers) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 155, in do response = retryable(self._perform)(method, File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 273, in _perform raise error from None databricks.sdk.errors.platform.ResourceDoesNotExist: Can't find a cluster policy with id: 001D9C7E67DB4BD9. 05:18 ERROR [databricks.labs.blueprint.parallel] installing components task failed: Timed out after 0:02:00 Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 540, in _create_database raise err File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 519, in _create_database deploy_schema(self._sql_backend, self._config.inventory_database) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 97, in deploy_schema deployer.deploy_schema() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/deployment.py", line 18, in deploy_schema self._sql_backend.execute(f"CREATE SCHEMA IF NOT EXISTS hive_metastore.{self._inventory_schema}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 221, in execute self._sql.run(sql) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/commands.py", line 113, in run ).result() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/_internal.py", line 67, in result return self._waiter(callback=callback, timeout=timeout, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/compute.py", line 7831, in wait_command_status_command_execution_finished_or_error poll = self.command_status(cluster_id=cluster_id, command_id=command_id, context_id=context_id) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/compute.py", line 7945, in command_status res = self._api.do('GET', '/api/1.2/commands/status', query=query, headers=headers) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 155, in do response = retryable(self._perform)(method, File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 273, in _perform raise error from None databricks.sdk.errors.platform.InternalError: TimeoutException: Timed out after 15 seconds The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 512, in _create_database_and_dashboards self._create_database() # Need the database before creating the dashboards File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 59, in wrapper raise TimeoutError(f'Timed out after {timeout}') from last_err TimeoutError: Timed out after 0:02:00 05:18 CRITICAL [databricks.labs.blueprint.parallel] All 'installing components' tasks failed!!! 05:18 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures 05:18 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000B797302611957 [group_name admins CAN_USE] -> [group_name ucx-lKCB-ra78a52fd7 CAN_USE] 05:18 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures 05:18 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000B797302611957') 05:18 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='000B797302611957') teardown: Can't find a cluster policy with id: 000B797302611957. 05:18 INFO [databricks.labs.ucx.install] Deleting UCX v0.36.1+420240916051514 from https://DATABRICKS_HOST 05:18 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_smuvn 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-txfe-ra78a52fd7@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-txfe-ra78a52fd7@example.com')], entitlements=[], external_id=None, groups=[], id='7321681649694012', name=Name(family_name=None, given_name='sdk-txfe-ra78a52fd7@example.com'), roles=[], schemas=[, ], user_name='sdk-txfe-ra78a52fd7@example.com') 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx-lKCB-ra78a52fd7', entitlements=[], external_id=None, groups=[], id='772554636059199', members=[ComplexValue(display='sdk-txfe-ra78a52fd7@example.com', primary=None, ref='Users/7321681649694012', type=None, value='7321681649694012')], meta=None, roles=[], schemas=[]) 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx-lKCB-ra78a52fd7', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='86070713970517', members=[ComplexValue(display='sdk-txfe-ra78a52fd7@example.com', primary=None, ref='Users/7321681649694012', type=None, value='7321681649694012')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 05:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smuvn', metastore_id=None, name='ucx_smuvn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) ```

_{Running from nightly #197}

❌ test_running_real_assessment_job_ext_hms: TimeoutError: timed out after 0:20:00: current status: CommandStatus.RUNNING (48m33.278s)

``` TimeoutError: timed out after 0:20:00: current status: CommandStatus.RUNNING TimeoutError: timed out after 0:20:00: current status: CommandStatus.QUEUED [gw0] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-rfbb-ra78a5309f@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-rfbb-ra78a5309f@example.com')], entitlements=[], external_id=None, groups=[], id='1163592680568323', name=Name(family_name=None, given_name='sdk-rfbb-ra78a5309f@example.com'), roles=[], schemas=[, ], user_name='sdk-rfbb-ra78a5309f@example.com') 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-KW5o-ra78a5309f: https://DATABRICKS_HOST#setting/accounts/groups/543151416692610 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-KW5o-ra78a5309f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='543151416692610', members=[ComplexValue(display='sdk-rfbb-ra78a5309f@example.com', primary=None, ref='Users/1163592680568323', type=None, value='1163592680568323')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-KW5o-ra78a5309f: https://accounts.CLOUD_ENVdatabricks.net/users/groups/576932995898432/members 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-KW5o-ra78a5309f', entitlements=[], external_id=None, groups=[], id='576932995898432', members=[ComplexValue(display='sdk-rfbb-ra78a5309f@example.com', primary=None, ref='Users/1163592680568323', type=None, value='1163592680568323')], meta=None, roles=[], schemas=[]) 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000E879319C40057 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000E879319C40057') 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000E879319C40057 [group_name admins CAN_USE] -> [group_name ucx-KW5o-ra78a5309f CAN_USE] 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_slesr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_slesr 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_slesr', metastore_id=None, name='ucx_slesr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 05:06 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.VVWp/config.yml) doesn't exist. 05:06 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 05:06 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 05:06 INFO [databricks.labs.ucx.install] Fetching installations... 05:07 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore 05:07 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 05:07 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:14 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:14 INFO [databricks.labs.ucx.install] Installing UCX v0.36.1+920240918051459 05:14 INFO [databricks.labs.ucx.install] Creating ucx schemas... 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 05:35 ERROR [databricks.labs.blueprint.parallel] installing components task failed: timed out after 0:20:00: current status: CommandStatus.RUNNING Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 516, in _create_database_and_dashboards self._create_database() # Need the database before creating the dashboards File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 544, in _create_database raise err File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 523, in _create_database deploy_schema(self._sql_backend, self._config.inventory_database) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 98, in deploy_schema deployer.deploy_schema() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/deployment.py", line 18, in deploy_schema self._sql_backend.execute(f"CREATE SCHEMA IF NOT EXISTS hive_metastore.{self._inventory_schema}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 221, in execute self._sql.run(sql) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/commands.py", line 113, in run ).result() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/_internal.py", line 67, in result return self._waiter(callback=callback, timeout=timeout, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/compute.py", line 7849, in wait_command_status_command_execution_finished_or_error raise TimeoutError(f'timed out after {timeout}: {status_message}') TimeoutError: timed out after 0:20:00: current status: CommandStatus.RUNNING 05:35 ERROR [databricks.labs.blueprint.parallel] More than half 'installing components' tasks failed: 0% results available (0/2). Took 0:20:11.517277 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-rfbb-ra78a5309f@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-rfbb-ra78a5309f@example.com')], entitlements=[], external_id=None, groups=[], id='1163592680568323', name=Name(family_name=None, given_name='sdk-rfbb-ra78a5309f@example.com'), roles=[], schemas=[, ], user_name='sdk-rfbb-ra78a5309f@example.com') 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-KW5o-ra78a5309f: https://DATABRICKS_HOST#setting/accounts/groups/543151416692610 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-KW5o-ra78a5309f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='543151416692610', members=[ComplexValue(display='sdk-rfbb-ra78a5309f@example.com', primary=None, ref='Users/1163592680568323', type=None, value='1163592680568323')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-KW5o-ra78a5309f: https://accounts.CLOUD_ENVdatabricks.net/users/groups/576932995898432/members 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-KW5o-ra78a5309f', entitlements=[], external_id=None, groups=[], id='576932995898432', members=[ComplexValue(display='sdk-rfbb-ra78a5309f@example.com', primary=None, ref='Users/1163592680568323', type=None, value='1163592680568323')], meta=None, roles=[], schemas=[]) 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000E879319C40057 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000E879319C40057') 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000E879319C40057 [group_name admins CAN_USE] -> [group_name ucx-KW5o-ra78a5309f CAN_USE] 05:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_slesr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_slesr 05:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_slesr', metastore_id=None, name='ucx_slesr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 05:06 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.VVWp/config.yml) doesn't exist. 05:06 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 05:06 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 05:06 INFO [databricks.labs.ucx.install] Fetching installations... 05:07 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore 05:07 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 05:07 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:14 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:14 INFO [databricks.labs.ucx.install] Installing UCX v0.36.1+920240918051459 05:14 INFO [databricks.labs.ucx.install] Creating ucx schemas... 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 05:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 05:35 ERROR [databricks.labs.blueprint.parallel] installing components task failed: timed out after 0:20:00: current status: CommandStatus.RUNNING Traceback (most recent call last): File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 516, in _create_database_and_dashboards self._create_database() # Need the database before creating the dashboards File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(*args, **kwargs) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 544, in _create_database raise err File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 523, in _create_database deploy_schema(self._sql_backend, self._config.inventory_database) File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/install.py", line 98, in deploy_schema deployer.deploy_schema() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/deployment.py", line 18, in deploy_schema self._sql_backend.execute(f"CREATE SCHEMA IF NOT EXISTS hive_metastore.{self._inventory_schema}") File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 221, in execute self._sql.run(sql) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/blueprint/commands.py", line 113, in run ).result() File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/_internal.py", line 67, in result return self._waiter(callback=callback, timeout=timeout, **kwargs) File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/compute.py", line 7849, in wait_command_status_command_execution_finished_or_error raise TimeoutError(f'timed out after {timeout}: {status_message}') TimeoutError: timed out after 0:20:00: current status: CommandStatus.RUNNING 05:35 ERROR [databricks.labs.blueprint.parallel] More than half 'installing components' tasks failed: 0% results available (0/2). Took 0:20:11.517277 05:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures 05:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000E879319C40057 [group_name admins CAN_USE] -> [group_name ucx-KW5o-ra78a5309f CAN_USE] 05:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures 05:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000E879319C40057') 05:35 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='000E879319C40057') teardown: Can't find a cluster policy with id: 000E879319C40057. 05:35 INFO [databricks.labs.ucx.install] Deleting UCX v0.36.1+920240918051459 from https://DATABRICKS_HOST 05:35 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_slesr 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-rfbb-ra78a5309f@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-rfbb-ra78a5309f@example.com')], entitlements=[], external_id=None, groups=[], id='1163592680568323', name=Name(family_name=None, given_name='sdk-rfbb-ra78a5309f@example.com'), roles=[], schemas=[, ], user_name='sdk-rfbb-ra78a5309f@example.com') 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx-KW5o-ra78a5309f', entitlements=[], external_id=None, groups=[], id='576932995898432', members=[ComplexValue(display='sdk-rfbb-ra78a5309f@example.com', primary=None, ref='Users/1163592680568323', type=None, value='1163592680568323')], meta=None, roles=[], schemas=[]) 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx-KW5o-ra78a5309f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='543151416692610', members=[ComplexValue(display='sdk-rfbb-ra78a5309f@example.com', primary=None, ref='Users/1163592680568323', type=None, value='1163592680568323')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 05:55 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_slesr', metastore_id=None, name='ucx_slesr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) ```

_{Running from nightly #199}

databrickslabs / ucx

Test failure: `test_running_real_assessment_job_ext_hms` #2638