Handle missing permission to read file

[JCZuurmond]

Handle missing permissions when reading files during linting

Linked issues

Resolves #1942 Partially resolves #1952

[github-actions[bot]]

❌ 195/197 passed, 2 flaky, 2 failed, 24 skipped, 4h44m39s total

❌ test_delete_ws_groups_should_delete_renamed_and_reflected_groups_only: TimeoutError: Timed out after 0:02:00 (3m55.641s)

``` TimeoutError: Timed out after 0:02:00 06:19 INFO [databricks.sdk] Using Databricks Metadata Service authentication 06:19 INFO [databricks.sdk] Using Databricks Metadata Service authentication 06:19 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sqz8l: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sqz8l 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sqz8l', metastore_id=None, name='ucx_sqz8l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 06:19 INFO [databricks.sdk] Using Databricks Metadata Service authentication 06:19 INFO [databricks.sdk] Using Databricks Metadata Service authentication 06:19 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sqz8l: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sqz8l 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sqz8l', metastore_id=None, name='ucx_sqz8l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-ehw644gp@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-ehw644gp@example.com')], entitlements=[], external_id=None, groups=[], id='6197215293514086', name=Name(family_name=None, given_name='sdk-ehw644gp@example.com'), roles=[], schemas=[, ], user_name='sdk-ehw644gp@example.com') 06:19 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_G5CEg: https://DATABRICKS_HOST#setting/accounts/groups/848022748261915 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_G5CEg', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='848022748261915', members=[ComplexValue(display='sdk-ehw644gp@example.com', primary=None, ref='Users/6197215293514086', type=None, value='6197215293514086')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 06:19 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_G5CEg: https://accounts.CLOUD_ENVdatabricks.net/users/groups/967123393493770/members 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_G5CEg', entitlements=[], external_id=None, groups=[], id='967123393493770', members=[ComplexValue(display='sdk-ehw644gp@example.com', primary=None, ref='Users/6197215293514086', type=None, value='6197215293514086')], meta=None, roles=[], schemas=[]) 06:19 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:19 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group 06:19 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 7 WorkspaceGroup 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] fetching groups inventory 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] crawling new batch for groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 7 WorkspaceGroup 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 3520 account groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Group listing provided, a subset of all groups will be migrated 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] found 1 new records for groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Renaming: ucx_G5CEg -> ucx-temp-ucx_G5CEg 06:20 INFO [databricks.labs.blueprint.parallel] rename groups in the workspace 1/1, rps: 0.257/sec 06:20 INFO [databricks.labs.blueprint.parallel] Finished 'rename groups in the workspace' tasks: 100% results available (1/1). Took 0:00:03.893643 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 3520 account groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] fetching groups inventory 06:20 INFO [databricks.labs.blueprint.parallel] reflect account groups on this workspace 1/1, rps: 1.064/sec 06:20 INFO [databricks.labs.blueprint.parallel] Finished 'reflect account groups on this workspace' tasks: 100% results available (1/1). Took 0:00:00.940189 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Found 8 WorkspaceGroup 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Found 58 Group 06:21 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] fetching groups inventory 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Skipping ucx_G5CEg: no longer in workspace 06:19 INFO [databricks.sdk] Using Databricks Metadata Service authentication 06:19 INFO [databricks.sdk] Using Databricks Metadata Service authentication 06:19 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sqz8l: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sqz8l 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sqz8l', metastore_id=None, name='ucx_sqz8l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-ehw644gp@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-ehw644gp@example.com')], entitlements=[], external_id=None, groups=[], id='6197215293514086', name=Name(family_name=None, given_name='sdk-ehw644gp@example.com'), roles=[], schemas=[, ], user_name='sdk-ehw644gp@example.com') 06:19 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_G5CEg: https://DATABRICKS_HOST#setting/accounts/groups/848022748261915 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_G5CEg', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='848022748261915', members=[ComplexValue(display='sdk-ehw644gp@example.com', primary=None, ref='Users/6197215293514086', type=None, value='6197215293514086')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 06:19 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_G5CEg: https://accounts.CLOUD_ENVdatabricks.net/users/groups/967123393493770/members 06:19 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_G5CEg', entitlements=[], external_id=None, groups=[], id='967123393493770', members=[ComplexValue(display='sdk-ehw644gp@example.com', primary=None, ref='Users/6197215293514086', type=None, value='6197215293514086')], meta=None, roles=[], schemas=[]) 06:19 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:19 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group 06:19 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 7 WorkspaceGroup 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] fetching groups inventory 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] crawling new batch for groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 7 WorkspaceGroup 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 3520 account groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Group listing provided, a subset of all groups will be migrated 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] found 1 new records for groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Renaming: ucx_G5CEg -> ucx-temp-ucx_G5CEg 06:20 INFO [databricks.labs.blueprint.parallel] rename groups in the workspace 1/1, rps: 0.257/sec 06:20 INFO [databricks.labs.blueprint.parallel] Finished 'rename groups in the workspace' tasks: 100% results available (1/1). Took 0:00:03.893643 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 3520 account groups 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group 06:20 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] fetching groups inventory 06:20 INFO [databricks.labs.blueprint.parallel] reflect account groups on this workspace 1/1, rps: 1.064/sec 06:20 INFO [databricks.labs.blueprint.parallel] Finished 'reflect account groups on this workspace' tasks: 100% results available (1/1). Took 0:00:00.940189 06:20 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Found 8 WorkspaceGroup 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Found 58 Group 06:21 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sqz8l.groups] fetching groups inventory 06:21 INFO [databricks.labs.ucx.workspace_access.groups] Skipping ucx_G5CEg: no longer in workspace 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sqz8l', metastore_id=None, name='ucx_sqz8l', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-ehw644gp@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-ehw644gp@example.com')], entitlements=[], external_id=None, groups=[], id='6197215293514086', name=Name(family_name=None, given_name='sdk-ehw644gp@example.com'), roles=[], schemas=[, ], user_name='sdk-ehw644gp@example.com') 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_G5CEg', entitlements=[], external_id=None, groups=[], id='967123393493770', members=[ComplexValue(display='sdk-ehw644gp@example.com', primary=None, ref='Users/6197215293514086', type=None, value='6197215293514086')], meta=None, roles=[], schemas=[]) 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 06:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_G5CEg', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='848022748261915', members=[ComplexValue(display='sdk-ehw644gp@example.com', primary=None, ref='Users/6197215293514086', type=None, value='6197215293514086')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) [gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

❌ test_running_real_migrate_groups_job: databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: Unknown: parse_logs: InternalError: verify group permissions('sdk-IoRH', [AclItem(principal='ucx_GXYI2', permission=), AclItem(principal='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', permission=)]) task failed: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 (7m50.583s)

``` databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: Unknown: parse_logs: InternalError: verify group permissions('sdk-IoRH', [AclItem(principal='ucx_GXYI2', permission=), AclItem(principal='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', permission=)]) task failed: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 124, in _verify_acls self._verify(scope_name, acl.principal, acl.permission) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/limiter.py", line 65, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 114, in _verify raise ValueError(msg) ValueError: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None, ValueError: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 [gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-oqbvi7sp@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-oqbvi7sp@example.com')], entitlements=[], external_id=None, groups=[], id='3597557589721096', name=Name(family_name=None, given_name='sdk-oqbvi7sp@example.com'), roles=[], schemas=[, ], user_name='sdk-oqbvi7sp@example.com') 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GXYI2: https://DATABRICKS_HOST#setting/accounts/groups/126649964945994 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GXYI2', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='126649964945994', members=[ComplexValue(display='sdk-oqbvi7sp@example.com', primary=None, ref='Users/3597557589721096', type=None, value='3597557589721096')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GXYI2: https://accounts.CLOUD_ENVdatabricks.net/users/groups/357253275519186/members 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GXYI2', entitlements=[], external_id=None, groups=[], id='357253275519186', members=[ComplexValue(display='sdk-oqbvi7sp@example.com', primary=None, ref='Users/3597557589721096', type=None, value='3597557589721096')], meta=None, roles=[], schemas=[]) 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001A3DAEDBC71F27 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001A3DAEDBC71F27') 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001A3DAEDBC71F27 [group_name admins CAN_USE] -> [group_name ucx_GXYI2 CAN_USE] 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srsrf: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srsrf 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srsrf', metastore_id=None, name='ucx_srsrf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_srsrf.ucx_twbds: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srsrf/ucx_twbds 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_srsrf.ucx_twbds', metastore_id=None, name='ucx_twbds', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070207'}, row_filter=None, schema_name='ucx_srsrf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_srsrf/ucx_twbds', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-IoRH 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-IoRH', 'ucx_GXYI2') 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s59mm: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s59mm 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s59mm', metastore_id=None, name='ucx_s59mm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:21 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/config.yml) doesn't exist. 06:21 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 06:21 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 06:21 INFO [databricks.labs.ucx.install] Fetching installations... 06:21 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 06:21 DEBUG [tests.integration.conftest] Waiting for clusters to start... 06:21 DEBUG [tests.integration.conftest] Waiting for clusters to start... 06:21 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+2120240702062145 06:21 INFO [databricks.labs.ucx.install] Creating ucx schemas... 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 06:22 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/README for the next steps. 06:22 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.grants] fetching grants inventory 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.grants] crawling new batch for grants 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.tables] fetching tables inventory 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.tables] crawling new batch for tables 06:22 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_srsrf] listing tables 06:22 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_srsrf.ucx_twbds] fetching table metadata 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.tables] found 1 new records for tables 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.udfs] fetching udfs inventory 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.udfs] crawling new batch for udfs 06:22 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_srsrf] listing udfs 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.udfs] found 0 new records for udfs 06:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.grants] found 4 new records for grants 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 4 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 4 06:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.permissions] found 4 new records for permissions 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Saved 4 to hive_metastore.ucx_s59mm.permissions 06:23 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-groups job: https://DATABRICKS_HOST#job/946793879389365 06:29 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS -------------- 06:29 INFO [databricks.labs.ucx:verify_metastore_attached] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/verify_metastore_attached.log 06:29 DEBUG [databricks.sdk:verify_metastore_attached] GET /api/2.1/unity-catalog/current-metastore-assignment < 200 OK < { < "TEST_SCHEMA_catalog_name": "hive_metastore", < "metastore_id": "8952c1e3-b265-4adf-98c3-6f755e2e1453", < "workspace_id": 7342989205138882 < } 06:29 INFO [databricks.labs.ucx:rename_workspace_local_groups] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/rename_workspace_local_groups.log 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (72 additional elements)" < ], < "itemsPerPage": 73, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 73 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=74&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 74, < "totalResults": 73 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 59 Group 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (72 additional elements)" < ], < "itemsPerPage": 73, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 73 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/60318222898965 < 200 OK < { < "displayName": "old_8snG", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "60318222898965", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/72057089919639 < 200 OK < { < "displayName": "ucx_GoL6u", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "72057089919639", < "members": [ < { < "$ref": "Users/1699065048952995", < "display": "sdk-p9tqsrt1@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/126649964945994 < 200 OK < { < "displayName": "ucx_GXYI2", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "126649964945994", < "members": [ < { < "$ref": "Users/3597557589721096", < "display": "sdk-oqbvi7sp@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/328597425812663 < 200 OK < { < "displayName": "old_ZeUo", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "328597425812663", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/390675359730119 < 200 OK < { < "displayName": "new_ZeUo", < "groups": [], < "id": "390675359730119", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/411450804401576 < 200 OK < { < "displayName": "sdk-5RvW", < "groups": [], < "id": "411450804401576", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/504030803486141 < 404 Not Found < { < "detail": "Group with id 504030803486141 not found.", < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:Error" < ], < "status": "404" < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/537502976576211 < 200 OK < { < "displayName": "ucxRnDH", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "537502976576211", < "members": [ < { < "$ref": "Users/2641331321724839", < "display": "sdk-xqkxb1ps@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/637469276499300 < 200 OK < { < "displayName": "ucx-temp-test_group_ucxnvAw", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "637469276499300", < "members": [ < { < "$ref": "Users/7467830011889775", < "display": "sdk-koxicumj@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/805641441398559 < 200 OK < { < "displayName": "old_Ma05", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "805641441398559", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/986314781892475 < 200 OK < { < "displayName": "new_8snG", < "groups": [], < "id": "986314781892475", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/1059849761235669 < 200 OK < { < "displayName": "new_Ma05", < "groups": [], < "id": "1059849761235669", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=74&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.blueprint.maintain", < "externalId": "8d8c87fa-4bae-43c3-adba-11c8d1a935ff", < "id": "1118462350168987", < "meta": { < "resourceType": "Group" < } < } < ], < "itemsPerPage": 1, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 74, < "totalResults": 74 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=75&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 75, < "totalResults": 72 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 11 WorkspaceGroup 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.ucx_s59mm.groups] fetching groups inventory 06:29 DEBUG [databricks.labs.lsql.backends:rename_workspace_local_groups] [spark][fetch] SELECT * FROM hive_metastore.ucx_s59mm.groups 06:29 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group ucx_GXYI2 defined in configuration does not exist on the groups table. Consider checking if the group exist in the workspace or re-running the assessment. 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.ucx_s59mm.groups] crawling new batch for groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (71 additional elements)" < ], < "itemsPerPage": 72, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 72 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/60318222898965 < 200 OK < { < "displayName": "old_8snG", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "60318222898965", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/72057089919639 < 200 OK < { < "displayName": "ucx_GoL6u", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "72057089919639", < "members": [ < { < "$ref": "Users/1699065048952995", < "display": "sdk-p9tqsrt1@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/82660694031426 < 200 OK < { < "displayName": "old_DUkb", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "82660694031426", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/126649964945994 < 200 OK < { < "displayName": "ucx_GXYI2", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "126649964945994", < "members": [ < { < "$ref": "Users/3597557589721096", < "display": "sdk-oqbvi7sp@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/328597425812663 < 200 OK < { < "displayName": "old_ZeUo", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "328597425812663", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/390675359730119 < 200 OK < { < "displayName": "new_ZeUo", < "groups": [], < "id": "390675359730119", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/480819444884023 < 200 OK < { < "displayName": "sdk-xOhp", < "groups": [], < "id": "480819444884023", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/537502976576211 < 200 OK < { < "displayName": "ucx-temp-ucxRnDH", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "537502976576211", < "members": [ < { < "$ref": "Users/2641331321724839", < "display": "sdk-xqkxb1ps@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/654490914933111 < 200 OK < { < "displayName": "new_DUkb", < "groups": [], < "id": "654490914933111", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/805641441398559 < 200 OK < { < "displayName": "old_Ma05", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "805641441398559", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/986314781892475 < 200 OK < { < "displayName": "new_8snG", < "groups": [], < "id": "986314781892475", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/1045730642338071 < 200 OK < { < "displayName": "sdk-40yD", < "groups": [], < "id": "1045730642338071", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/1059849761235669 < 200 OK < { < "displayName": "new_Ma05", < "groups": [], < "id": "1059849761235669", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=73&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 73, < "totalResults": 72 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 13 WorkspaceGroup 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing account groups with id,displayName,externalId... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "ucx_EMQk", < "id": "747915403144" < }, < { < "displayName": "rename-LFcF-ucx_GlGZb", < "id": "839447153138" < }, < "... (3516 additional elements)" < ], < "itemsPerPage": 3518, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 3518 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 3517 account groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group listing provided, a subset of all groups will be migrated 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.ucx_s59mm.groups] found 1 new records for groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Renaming: ucx_GXYI2 -> rename-mSvT-ucx_GXYI2 06:29 DEBUG [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Starting 1 tasks in 8 threads 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] PATCH /api/2.0/preview/scim/v2/Groups/126649964945994 > { > "Operations": [ > { > "op": "replace", > "path": "displayName", > "value": "**REDACTED**" > } > ] > } < 200 OK < { < "displayName": "rename-mSvT-ucx_GXYI2", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "126649964945994", < "members": [ < { < "$ref": "Users/3597557589721096", < "display": "sdk-oqbvi7sp@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] rename groups in the workspace 1/1, rps: 0.299/sec 06:29 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Finished 'rename groups in the workspace' tasks: 100% results available (1/1). Took 0:00:03.353502 06:29 INFO [databricks.labs.ucx:reflect_account_groups_on_workspace] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/reflect_account_groups_on_workspace.log 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing account groups with id,displayName,externalId... 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "ucx_EMQk", < "id": "747915403144" < }, < { < "displayName": "rename-LFcF-ucx_GlGZb", < "id": "839447153138" < }, < "... (3513 additional elements)" < ], < "itemsPerPage": 3515, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 3515 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 3514 account groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 57 Group 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:reflect_account_groups_on_workspace] [hive_metastore.ucx_s59mm.groups] fetching groups inventory 06:29 DEBUG [databricks.labs.lsql.backends:reflect_account_groups_on_workspace] [spark][fetch] SELECT * FROM hive_metastore.ucx_s59mm.groups 06:29 DEBUG [databricks.labs.blueprint.parallel:reflect_account_groups_on_workspace] Starting 1 tasks in 8 threads 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] PUT /api/2.0/preview/permissionassignments/principals/357253275519186 > [raw stream] < 200 OK < { < "permissions": [ < "USER" < ], < "principal": { < "display_name": "ucx_GXYI2", < "group_name": "ucx_GXYI2", < "principal_id": 357253275519186 < } < } 06:29 INFO [databricks.labs.blueprint.parallel:reflect_account_groups_on_workspace] reflect account groups on this workspace 1/1, rps: 1.107/sec 06:29 INFO [databricks.labs.blueprint.parallel:reflect_account_groups_on_workspace] Finished 'reflect account groups on this workspace' tasks: 100% results available (1/1). Took 0:00:00.912199 06:29 INFO [databricks.labs.ucx:apply_permissions_to_account_groups] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/apply_permissions_to_account_groups.log 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:apply_permissions_to_account_groups] [hive_metastore.ucx_s59mm.groups] fetching groups inventory 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SELECT * FROM hive_metastore.ucx_s59mm.groups 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (62 additional elements)" < ], < "itemsPerPage": 63, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 63 < } 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=64&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 64, < "totalResults": 63 < } 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Loading inventory table hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SELECT COUNT(*) as cnt FROM hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SELECT object_id, object_type, raw FROM hive_metastore.ucx_s59mm.permissions 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Applying the permissions to account groups. Total groups to apply permissions: 1. Total permissions found: 4 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Total tasks for TABLE: 1 06:29 DEBUG [databricks.labs.ucx.workspace_access.generic:apply_permissions_to_account_groups] Skipping AccessControlResponse(all_permissions=[Permission(inherited=True, inherited_from_object=['/cluster-policies/cluster-policies'], permission_level=)], display_name=None, group_name='admins', service_principal_name=None, user_name=None) for cluster-policy//cluster-policies/001A3DAEDBC71F27 because it is not in scope 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Total tasks for cluster-policies: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Total tasks for secrets: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Starting to apply permissions on account groups. Total tasks: 3 06:29 DEBUG [databricks.labs.blueprint.parallel:apply_permissions_to_account_groups] Starting 3 tasks in 8 threads 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][execute] GRANT SELECT ON TABLE hive_metastore.ucx_srsrf.ucx_twbds TO `ucx_GXYI2` 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] POST /api/2.0/secrets/acls/put > { > "permission": "WRITE", > "principal": "ucx_GXYI2", > "scope": "sdk-IoRH" > } < 200 OK < {} 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/secrets/acls/list?scope=sdk-IoRH < 200 OK < { < "items": [ < { < "permission": "WRITE", < "principal": "ucx_GXYI2" < }, < { < "permission": "MANAGE", < "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } < ] < } 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] POST /api/2.0/secrets/acls/put > { > "permission": "MANAGE", > "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d", > "scope": "sdk-IoRH" > } < 200 OK < {} 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/secrets/acls/list?scope=sdk-IoRH < 200 OK < { < "items": [ < { < "permission": "WRITE", < "principal": "rename-mSvT-ucx_GXYI2" < }, < { < "permission": "MANAGE", < "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } < ] < } 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SHOW GRANTS ON TABLE hive_metastore.ucx_srsrf.ucx_twbds 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] PATCH /api/2.0/permissions/cluster-policies/001A3DAEDBC71F27 > { > "access_control_list": [ > { > "group_name": "ucx_GXYI2", > "permission_level": "CAN_USE" > } > ] > } < 200 OK < { < "access_control_list": [ < { < "all_permissions": [ < { < "inherited": false, < "permission_level": "CAN_USE" < } < ], < "display_name": "labs-account-admin-identity", < "service_principal_name": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < }, < "... (3 additional elements)" < ], < "object_id": "/cluster-policies/001A3DAEDBC71F27", < "object_type": "cluster-policy" < } 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/permissions/cluster-policies/001A3DAEDBC71F27 < 200 OK < { < "access_control_list": [ < { < "all_permissions": [ < { < "inherited": false, < "permission_level": "CAN_USE" < } < ], < "display_name": "labs-account-admin-identity", < "service_principal_name": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < }, < "... (3 additional elements)" < ], < "object_id": "/cluster-policies/001A3DAEDBC71F27", < "object_type": "cluster-policy" < } 06:29 INFO [databricks.labs.blueprint.parallel:apply_permissions_to_account_groups] apply account group permissions 3/3, rps: 1.237/sec 06:29 INFO [databricks.labs.blueprint.parallel:apply_permissions_to_account_groups] Finished 'apply account group permissions' tasks: 100% results available (3/3). Took 0:00:02.432121 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Permissions were applied 06:29 INFO [databricks.labs.ucx:validate_groups_permissions] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/validate_groups_permissions.log 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (62 additional elements)" < ], < "itemsPerPage": 63, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 63 < } 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=64&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 64, < "totalResults": 63 < } 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Loading inventory table hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SELECT COUNT(*) as cnt FROM hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SELECT object_id, object_type, raw FROM hive_metastore.ucx_s59mm.permissions 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total permissions found: 4 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total tasks for TABLE: 2 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total tasks for cluster-policies: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total tasks for secrets: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Starting to verify permissions. Total tasks: 4 06:29 DEBUG [databricks.labs.blueprint.parallel:validate_groups_permissions] Starting 4 tasks in 8 threads 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SHOW GRANTS ON TABLE hive_metastore.ucx_srsrf.ucx_twbds 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SHOW GRANTS ON TABLE hive_metastore.ucx_srsrf.ucx_twbds 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/secrets/acls/list?scope=sdk-IoRH < 200 OK < { < "items": [ < { < "permission": "WRITE", < "principal": "rename-mSvT-ucx_GXYI2" < }, < { < "permission": "MANAGE", < "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } < ] < } 06:29 ERROR [databricks.labs.blueprint.parallel:validate_groups_permissions] verify group permissions('sdk-IoRH', [AclItem(principal='ucx_GXYI2', permission=), AclItem(principal='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', permission=)]) task failed: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 124, in _verify_acls self._verify(scope_name, acl.principal, acl.permission) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/limiter.py", line 65, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 114, in _verify raise ValueError(msg) ValueError: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/permissions/cluster-policies/001A3DAEDBC71F27 < 200 OK < { < "access_control_list": [ < { < "all_permissions": [ < { < "inherited": false, < "permission_level": "CAN_USE" < } < ], < "display_name": "labs-account-admin-identity", < "service_principal_name": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < }, < "... (3 additional elements)" < ], < "object_id": "/cluster-policies/001A3DAEDBC71F27", < "object_type": "cluster-policy" < } 06:29 INFO [databricks.labs.blueprint.parallel:validate_groups_permissions] verify group permissions 4/4, rps: 2.432/sec 06:29 WARNING [databricks.labs.blueprint.parallel:validate_groups_permissions] Some 'verify group permissions' tasks failed: 75% results available (3/4). Took 0:00:01.651300 06:29 ERROR [databricks.labs.ucx:validate_groups_permissions] Execute `databricks workspace export //Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/validate_groups_permissions.log` locally to troubleshoot with more details. Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None 06:29 DEBUG [databricks:validate_groups_permissions] Task crash details Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/runtime.py", line 96, in trigger current_task(ctx) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/workflows.py", line 50, in validate_groups_permissions ctx.permission_manager.verify_group_permissions() File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/manager.py", line 117, in verify_group_permissions Threads.strict("verify group permissions", verifier_tasks) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/parallel.py", line 62, in strict raise errs[0] File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 124, in _verify_acls self._verify(scope_name, acl.principal, acl.permission) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/limiter.py", line 65, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 114, in _verify raise ValueError(msg) ValueError: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None 06:29 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ---------- 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-oqbvi7sp@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-oqbvi7sp@example.com')], entitlements=[], external_id=None, groups=[], id='3597557589721096', name=Name(family_name=None, given_name='sdk-oqbvi7sp@example.com'), roles=[], schemas=[, ], user_name='sdk-oqbvi7sp@example.com') 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GXYI2: https://DATABRICKS_HOST#setting/accounts/groups/126649964945994 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GXYI2', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='126649964945994', members=[ComplexValue(display='sdk-oqbvi7sp@example.com', primary=None, ref='Users/3597557589721096', type=None, value='3597557589721096')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GXYI2: https://accounts.CLOUD_ENVdatabricks.net/users/groups/357253275519186/members 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GXYI2', entitlements=[], external_id=None, groups=[], id='357253275519186', members=[ComplexValue(display='sdk-oqbvi7sp@example.com', primary=None, ref='Users/3597557589721096', type=None, value='3597557589721096')], meta=None, roles=[], schemas=[]) 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001A3DAEDBC71F27 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001A3DAEDBC71F27') 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001A3DAEDBC71F27 [group_name admins CAN_USE] -> [group_name ucx_GXYI2 CAN_USE] 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srsrf: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srsrf 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srsrf', metastore_id=None, name='ucx_srsrf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_srsrf.ucx_twbds: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srsrf/ucx_twbds 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_srsrf.ucx_twbds', metastore_id=None, name='ucx_twbds', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070207'}, row_filter=None, schema_name='ucx_srsrf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_srsrf/ucx_twbds', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-IoRH 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-IoRH', 'ucx_GXYI2') 06:21 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s59mm: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s59mm 06:21 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s59mm', metastore_id=None, name='ucx_s59mm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:21 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/config.yml) doesn't exist. 06:21 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 06:21 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 06:21 INFO [databricks.labs.ucx.install] Fetching installations... 06:21 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 06:21 DEBUG [tests.integration.conftest] Waiting for clusters to start... 06:21 DEBUG [tests.integration.conftest] Waiting for clusters to start... 06:21 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+2120240702062145 06:21 INFO [databricks.labs.ucx.install] Creating ucx schemas... 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 06:22 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 06:22 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/README for the next steps. 06:22 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.grants] fetching grants inventory 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.grants] crawling new batch for grants 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.tables] fetching tables inventory 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.tables] crawling new batch for tables 06:22 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_srsrf] listing tables 06:22 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_srsrf.ucx_twbds] fetching table metadata 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.tables] found 1 new records for tables 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.udfs] fetching udfs inventory 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.udfs] crawling new batch for udfs 06:22 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_srsrf] listing udfs 06:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.udfs] found 0 new records for udfs 06:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.grants] found 4 new records for grants 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 4 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 4 06:23 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s59mm.permissions] found 4 new records for permissions 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table 06:23 INFO [databricks.labs.ucx.workspace_access.manager] Saved 4 to hive_metastore.ucx_s59mm.permissions 06:23 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-groups job: https://DATABRICKS_HOST#job/946793879389365 06:29 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS -------------- 06:29 INFO [databricks.labs.ucx:verify_metastore_attached] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/verify_metastore_attached.log 06:29 DEBUG [databricks.sdk:verify_metastore_attached] GET /api/2.1/unity-catalog/current-metastore-assignment < 200 OK < { < "TEST_SCHEMA_catalog_name": "hive_metastore", < "metastore_id": "8952c1e3-b265-4adf-98c3-6f755e2e1453", < "workspace_id": 7342989205138882 < } 06:29 INFO [databricks.labs.ucx:rename_workspace_local_groups] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/rename_workspace_local_groups.log 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (72 additional elements)" < ], < "itemsPerPage": 73, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 73 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=74&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 74, < "totalResults": 73 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 59 Group 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (72 additional elements)" < ], < "itemsPerPage": 73, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 73 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/60318222898965 < 200 OK < { < "displayName": "old_8snG", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "60318222898965", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/72057089919639 < 200 OK < { < "displayName": "ucx_GoL6u", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "72057089919639", < "members": [ < { < "$ref": "Users/1699065048952995", < "display": "sdk-p9tqsrt1@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/126649964945994 < 200 OK < { < "displayName": "ucx_GXYI2", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "126649964945994", < "members": [ < { < "$ref": "Users/3597557589721096", < "display": "sdk-oqbvi7sp@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/328597425812663 < 200 OK < { < "displayName": "old_ZeUo", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "328597425812663", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/390675359730119 < 200 OK < { < "displayName": "new_ZeUo", < "groups": [], < "id": "390675359730119", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/411450804401576 < 200 OK < { < "displayName": "sdk-5RvW", < "groups": [], < "id": "411450804401576", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/504030803486141 < 404 Not Found < { < "detail": "Group with id 504030803486141 not found.", < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:Error" < ], < "status": "404" < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/537502976576211 < 200 OK < { < "displayName": "ucxRnDH", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "537502976576211", < "members": [ < { < "$ref": "Users/2641331321724839", < "display": "sdk-xqkxb1ps@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/637469276499300 < 200 OK < { < "displayName": "ucx-temp-test_group_ucxnvAw", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "637469276499300", < "members": [ < { < "$ref": "Users/7467830011889775", < "display": "sdk-koxicumj@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/805641441398559 < 200 OK < { < "displayName": "old_Ma05", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "805641441398559", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/986314781892475 < 200 OK < { < "displayName": "new_8snG", < "groups": [], < "id": "986314781892475", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/1059849761235669 < 200 OK < { < "displayName": "new_Ma05", < "groups": [], < "id": "1059849761235669", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=74&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.blueprint.maintain", < "externalId": "8d8c87fa-4bae-43c3-adba-11c8d1a935ff", < "id": "1118462350168987", < "meta": { < "resourceType": "Group" < } < } < ], < "itemsPerPage": 1, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 74, < "totalResults": 74 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=75&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 75, < "totalResults": 72 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 11 WorkspaceGroup 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.ucx_s59mm.groups] fetching groups inventory 06:29 DEBUG [databricks.labs.lsql.backends:rename_workspace_local_groups] [spark][fetch] SELECT * FROM hive_metastore.ucx_s59mm.groups 06:29 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group ucx_GXYI2 defined in configuration does not exist on the groups table. Consider checking if the group exist in the workspace or re-running the assessment. 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.ucx_s59mm.groups] crawling new batch for groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (71 additional elements)" < ], < "itemsPerPage": 72, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 72 < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/60318222898965 < 200 OK < { < "displayName": "old_8snG", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "60318222898965", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/72057089919639 < 200 OK < { < "displayName": "ucx_GoL6u", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "72057089919639", < "members": [ < { < "$ref": "Users/1699065048952995", < "display": "sdk-p9tqsrt1@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/82660694031426 < 200 OK < { < "displayName": "old_DUkb", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "82660694031426", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/126649964945994 < 200 OK < { < "displayName": "ucx_GXYI2", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "126649964945994", < "members": [ < { < "$ref": "Users/3597557589721096", < "display": "sdk-oqbvi7sp@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/328597425812663 < 200 OK < { < "displayName": "old_ZeUo", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "328597425812663", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/390675359730119 < 200 OK < { < "displayName": "new_ZeUo", < "groups": [], < "id": "390675359730119", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/480819444884023 < 200 OK < { < "displayName": "sdk-xOhp", < "groups": [], < "id": "480819444884023", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/537502976576211 < 200 OK < { < "displayName": "ucx-temp-ucxRnDH", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "537502976576211", < "members": [ < { < "$ref": "Users/2641331321724839", < "display": "sdk-xqkxb1ps@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/654490914933111 < 200 OK < { < "displayName": "new_DUkb", < "groups": [], < "id": "654490914933111", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/805641441398559 < 200 OK < { < "displayName": "old_Ma05", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "805641441398559", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/986314781892475 < 200 OK < { < "displayName": "new_8snG", < "groups": [], < "id": "986314781892475", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/1045730642338071 < 200 OK < { < "displayName": "sdk-40yD", < "groups": [], < "id": "1045730642338071", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/1059849761235669 < 200 OK < { < "displayName": "new_Ma05", < "groups": [], < "id": "1059849761235669", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,externalId,roles,entitlements&startIndex=73&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 73, < "totalResults": 72 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 13 WorkspaceGroup 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing account groups with id,displayName,externalId... 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "ucx_EMQk", < "id": "747915403144" < }, < { < "displayName": "rename-LFcF-ucx_GlGZb", < "id": "839447153138" < }, < "... (3516 additional elements)" < ], < "itemsPerPage": 3518, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 3518 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 3517 account groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group listing provided, a subset of all groups will be migrated 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.ucx_s59mm.groups] found 1 new records for groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Renaming: ucx_GXYI2 -> rename-mSvT-ucx_GXYI2 06:29 DEBUG [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Starting 1 tasks in 8 threads 06:29 DEBUG [databricks.sdk:rename_workspace_local_groups] PATCH /api/2.0/preview/scim/v2/Groups/126649964945994 > { > "Operations": [ > { > "op": "replace", > "path": "displayName", > "value": "**REDACTED**" > } > ] > } < 200 OK < { < "displayName": "rename-mSvT-ucx_GXYI2", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "126649964945994", < "members": [ < { < "$ref": "Users/3597557589721096", < "display": "sdk-oqbvi7sp@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 06:29 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] rename groups in the workspace 1/1, rps: 0.299/sec 06:29 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Finished 'rename groups in the workspace' tasks: 100% results available (1/1). Took 0:00:03.353502 06:29 INFO [databricks.labs.ucx:reflect_account_groups_on_workspace] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/reflect_account_groups_on_workspace.log 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing account groups with id,displayName,externalId... 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "ucx_EMQk", < "id": "747915403144" < }, < { < "displayName": "rename-LFcF-ucx_GlGZb", < "id": "839447153138" < }, < "... (3513 additional elements)" < ], < "itemsPerPage": 3515, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 3515 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 3514 account groups 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta... 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 06:29 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 57 Group 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:reflect_account_groups_on_workspace] [hive_metastore.ucx_s59mm.groups] fetching groups inventory 06:29 DEBUG [databricks.labs.lsql.backends:reflect_account_groups_on_workspace] [spark][fetch] SELECT * FROM hive_metastore.ucx_s59mm.groups 06:29 DEBUG [databricks.labs.blueprint.parallel:reflect_account_groups_on_workspace] Starting 1 tasks in 8 threads 06:29 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] PUT /api/2.0/preview/permissionassignments/principals/357253275519186 > [raw stream] < 200 OK < { < "permissions": [ < "USER" < ], < "principal": { < "display_name": "ucx_GXYI2", < "group_name": "ucx_GXYI2", < "principal_id": 357253275519186 < } < } 06:29 INFO [databricks.labs.blueprint.parallel:reflect_account_groups_on_workspace] reflect account groups on this workspace 1/1, rps: 1.107/sec 06:29 INFO [databricks.labs.blueprint.parallel:reflect_account_groups_on_workspace] Finished 'reflect account groups on this workspace' tasks: 100% results available (1/1). Took 0:00:00.912199 06:29 INFO [databricks.labs.ucx:apply_permissions_to_account_groups] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/apply_permissions_to_account_groups.log 06:29 DEBUG [databricks.labs.ucx.framework.crawlers:apply_permissions_to_account_groups] [hive_metastore.ucx_s59mm.groups] fetching groups inventory 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SELECT * FROM hive_metastore.ucx_s59mm.groups 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (62 additional elements)" < ], < "itemsPerPage": 63, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 63 < } 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=64&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 64, < "totalResults": 63 < } 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Loading inventory table hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SELECT COUNT(*) as cnt FROM hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SELECT object_id, object_type, raw FROM hive_metastore.ucx_s59mm.permissions 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Applying the permissions to account groups. Total groups to apply permissions: 1. Total permissions found: 4 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Total tasks for TABLE: 1 06:29 DEBUG [databricks.labs.ucx.workspace_access.generic:apply_permissions_to_account_groups] Skipping AccessControlResponse(all_permissions=[Permission(inherited=True, inherited_from_object=['/cluster-policies/cluster-policies'], permission_level=)], display_name=None, group_name='admins', service_principal_name=None, user_name=None) for cluster-policy//cluster-policies/001A3DAEDBC71F27 because it is not in scope 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Total tasks for cluster-policies: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Total tasks for secrets: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Starting to apply permissions on account groups. Total tasks: 3 06:29 DEBUG [databricks.labs.blueprint.parallel:apply_permissions_to_account_groups] Starting 3 tasks in 8 threads 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][execute] GRANT SELECT ON TABLE hive_metastore.ucx_srsrf.ucx_twbds TO `ucx_GXYI2` 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] POST /api/2.0/secrets/acls/put > { > "permission": "WRITE", > "principal": "ucx_GXYI2", > "scope": "sdk-IoRH" > } < 200 OK < {} 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/secrets/acls/list?scope=sdk-IoRH < 200 OK < { < "items": [ < { < "permission": "WRITE", < "principal": "ucx_GXYI2" < }, < { < "permission": "MANAGE", < "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } < ] < } 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] POST /api/2.0/secrets/acls/put > { > "permission": "MANAGE", > "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d", > "scope": "sdk-IoRH" > } < 200 OK < {} 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/secrets/acls/list?scope=sdk-IoRH < 200 OK < { < "items": [ < { < "permission": "WRITE", < "principal": "rename-mSvT-ucx_GXYI2" < }, < { < "permission": "MANAGE", < "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } < ] < } 06:29 DEBUG [databricks.labs.lsql.backends:apply_permissions_to_account_groups] [spark][fetch] SHOW GRANTS ON TABLE hive_metastore.ucx_srsrf.ucx_twbds 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] PATCH /api/2.0/permissions/cluster-policies/001A3DAEDBC71F27 > { > "access_control_list": [ > { > "group_name": "ucx_GXYI2", > "permission_level": "CAN_USE" > } > ] > } < 200 OK < { < "access_control_list": [ < { < "all_permissions": [ < { < "inherited": false, < "permission_level": "CAN_USE" < } < ], < "display_name": "labs-account-admin-identity", < "service_principal_name": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < }, < "... (3 additional elements)" < ], < "object_id": "/cluster-policies/001A3DAEDBC71F27", < "object_type": "cluster-policy" < } 06:29 DEBUG [databricks.sdk:apply_permissions_to_account_groups] GET /api/2.0/permissions/cluster-policies/001A3DAEDBC71F27 < 200 OK < { < "access_control_list": [ < { < "all_permissions": [ < { < "inherited": false, < "permission_level": "CAN_USE" < } < ], < "display_name": "labs-account-admin-identity", < "service_principal_name": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < }, < "... (3 additional elements)" < ], < "object_id": "/cluster-policies/001A3DAEDBC71F27", < "object_type": "cluster-policy" < } 06:29 INFO [databricks.labs.blueprint.parallel:apply_permissions_to_account_groups] apply account group permissions 3/3, rps: 1.237/sec 06:29 INFO [databricks.labs.blueprint.parallel:apply_permissions_to_account_groups] Finished 'apply account group permissions' tasks: 100% results available (3/3). Took 0:00:02.432121 06:29 INFO [databricks.labs.ucx.workspace_access.manager:apply_permissions_to_account_groups] Permissions were applied 06:29 INFO [databricks.labs.ucx:validate_groups_permissions] UCX v0.27.2+2120240702062145 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/validate_groups_permissions.log 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (62 additional elements)" < ], < "itemsPerPage": 63, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 63 < } 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=64&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 64, < "totalResults": 63 < } 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Loading inventory table hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SELECT COUNT(*) as cnt FROM hive_metastore.ucx_s59mm.permissions 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SELECT object_id, object_type, raw FROM hive_metastore.ucx_s59mm.permissions 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total permissions found: 4 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total tasks for TABLE: 2 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total tasks for cluster-policies: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Total tasks for secrets: 1 06:29 INFO [databricks.labs.ucx.workspace_access.manager:validate_groups_permissions] Starting to verify permissions. Total tasks: 4 06:29 DEBUG [databricks.labs.blueprint.parallel:validate_groups_permissions] Starting 4 tasks in 8 threads 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SHOW GRANTS ON TABLE hive_metastore.ucx_srsrf.ucx_twbds 06:29 DEBUG [databricks.labs.lsql.backends:validate_groups_permissions] [spark][fetch] SHOW GRANTS ON TABLE hive_metastore.ucx_srsrf.ucx_twbds 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/secrets/acls/list?scope=sdk-IoRH < 200 OK < { < "items": [ < { < "permission": "WRITE", < "principal": "rename-mSvT-ucx_GXYI2" < }, < { < "permission": "MANAGE", < "principal": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } < ] < } 06:29 ERROR [databricks.labs.blueprint.parallel:validate_groups_permissions] verify group permissions('sdk-IoRH', [AclItem(principal='ucx_GXYI2', permission=), AclItem(principal='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', permission=)]) task failed: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 124, in _verify_acls self._verify(scope_name, acl.principal, acl.permission) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/limiter.py", line 65, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 114, in _verify raise ValueError(msg) ValueError: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None 06:29 DEBUG [databricks.sdk:validate_groups_permissions] GET /api/2.0/permissions/cluster-policies/001A3DAEDBC71F27 < 200 OK < { < "access_control_list": [ < { < "all_permissions": [ < { < "inherited": false, < "permission_level": "CAN_USE" < } < ], < "display_name": "labs-account-admin-identity", < "service_principal_name": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < }, < "... (3 additional elements)" < ], < "object_id": "/cluster-policies/001A3DAEDBC71F27", < "object_type": "cluster-policy" < } 06:29 INFO [databricks.labs.blueprint.parallel:validate_groups_permissions] verify group permissions 4/4, rps: 2.432/sec 06:29 WARNING [databricks.labs.blueprint.parallel:validate_groups_permissions] Some 'verify group permissions' tasks failed: 75% results available (3/4). Took 0:00:01.651300 06:29 ERROR [databricks.labs.ucx:validate_groups_permissions] Execute `databricks workspace export //Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.mSvT/logs/migrate-groups/run-948814237918560-0/validate_groups_permissions.log` locally to troubleshoot with more details. Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None 06:29 DEBUG [databricks:validate_groups_permissions] Task crash details Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/runtime.py", line 96, in trigger current_task(ctx) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/workflows.py", line 50, in validate_groups_permissions ctx.permission_manager.verify_group_permissions() File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/manager.py", line 117, in verify_group_permissions Threads.strict("verify group permissions", verifier_tasks) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/parallel.py", line 62, in strict raise errs[0] File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner return func(*args, **kwargs), None ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 124, in _verify_acls self._verify(scope_name, acl.principal, acl.permission) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/blueprint/limiter.py", line 65, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2fe917ad-888a-4753-a478-62b7c721b241/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/secrets.py", line 114, in _verify raise ValueError(msg) ValueError: Couldn't find permission for scope sdk-IoRH and group ucx_GXYI2 acl to be applied=AclPermission.WRITE acl found in the object=None 06:29 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ---------- 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope acl fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope acl fixture: ('sdk-IoRH', 'ucx_GXYI2') 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while secret scope acl ('sdk-IoRH', 'ucx_GXYI2') teardown: ACL for ucx_GXYI2 does not exist in scope sdk-IoRH! 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope fixture: sdk-IoRH 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 001A3DAEDBC71F27 [group_name admins CAN_USE] -> [group_name ucx_GXYI2 CAN_USE] 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='001A3DAEDBC71F27') 06:29 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+2120240702062145 from https://DATABRICKS_HOST 06:29 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s59mm 06:29 INFO [databricks.labs.ucx.install] Deleting jobs 06:29 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1044513468642094. 06:29 INFO [databricks.labs.ucx.install] Deleting assessment job_id=492168431851983. 06:29 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=560502794385505. 06:29 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=86248866298010. 06:29 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=946793879389365. 06:29 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=418417124696054. 06:29 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=195244619139067. 06:29 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=199463573227540. 06:29 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=213264995628021. 06:29 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=397521740913487. 06:29 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=850521732908906. 06:29 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=18254585833548. 06:29 INFO [databricks.labs.ucx.install] Deleting failing job_id=1078129763970136. 06:29 INFO [databricks.labs.ucx.install] Deleting cluster policy 06:29 INFO [databricks.labs.ucx.install] Deleting secret scope 06:29 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-oqbvi7sp@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-oqbvi7sp@example.com')], entitlements=[], external_id=None, groups=[], id='3597557589721096', name=Name(family_name=None, given_name='sdk-oqbvi7sp@example.com'), roles=[], schemas=[, ], user_name='sdk-oqbvi7sp@example.com') 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_GXYI2', entitlements=[], external_id=None, groups=[], id='357253275519186', members=[ComplexValue(display='sdk-oqbvi7sp@example.com', primary=None, ref='Users/3597557589721096', type=None, value='3597557589721096')], meta=None, roles=[], schemas=[]) 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_GXYI2', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='126649964945994', members=[ComplexValue(display='sdk-oqbvi7sp@example.com', primary=None, ref='Users/3597557589721096', type=None, value='3597557589721096')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 table fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_srsrf.ucx_twbds', metastore_id=None, name='ucx_twbds', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070207'}, row_filter=None, schema_name='ucx_srsrf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_srsrf/ucx_twbds', table_constraints=None, table_id=None, table_type=, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None) 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 2 schema fixtures 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srsrf', metastore_id=None, name='ucx_srsrf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 06:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s59mm', metastore_id=None, name='ucx_s59mm', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

Flaky tests:

• 🤪 test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas] (16m18.84s)
• 🤪 test_table_migration_job_refreshes_migration_status[regular-migrate-tables] (9m10.017s)

_{Running from acceptance #4082}