Test failure: `test_experimental_permissions_migration_for_group_with_same_name`

❌ test_experimental_permissions_migration_for_group_with_same_name: databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: InternalError: Skipping ucx-50JJ-ra78a5548f: group already exists in workspace, NotFound: Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace (3m34.814s)

``` ... (skipped 7425 bytes) rams:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 57 Group 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/448371539366696 < 200 OK < { < "displayName": "sdk-cdeA-ra78a5548f", < "groups": [], < "id": "448371539366696", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/824705871265106 < 200 OK < { < "displayName": "sdk-jftY-ra78a5548f", < "groups": [], < "id": "824705871265106", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 3 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.dummy_sicwj.groups] fetching groups inventory 05:45 DEBUG [databricks.labs.lsql.backends:rename_workspace_local_groups] [spark][fetch] SELECT * FROM `hive_metastore`.`dummy_sicwj`.`groups` 05:45 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group ucx-50JJ-ra78a5548f defined in configuration does not exist on the groups table. Consider checking if the group exist in the workspace or re-running the assessment. 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.dummy_sicwj.groups] crawling new set of snapshot data for groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/448371539366696 < 200 OK < { < "displayName": "sdk-cdeA-ra78a5548f", < "groups": [], < "id": "448371539366696", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/824705871265106 < 200 OK < { < "displayName": "sdk-jftY-ra78a5548f", < "groups": [], < "id": "824705871265106", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 3 WorkspaceGroup 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing account groups with id,displayName,externalId... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "role.github-emu.field-eng.labs-infra-write", < "externalId": "d8953329-3875-4ff2-bede-49cc584a1fbb", < "id": "14154893514742" < }, < "... (69 additional elements)" < ], < "itemsPerPage": 70, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 70 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 69 account groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group listing provided, a subset of all groups will be migrated 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.dummy_sicwj.groups] found 1 new records for groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Starting to rename 1 groups for migration... 05:45 DEBUG [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Starting 1 tasks in 8 threads 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Renaming group: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] PATCH /api/2.0/preview/scim/v2/Groups/726451466196921 > { > "Operations": [ > { > "op": "replace", > "path": "displayName", > "value": "**REDACTED**" > } > ] > } < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] rename groups in the workspace 1/1, rps: 0.359/sec 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Finished 'rename groups in the workspace' tasks: 0% results available (0/1). Took 0:00:02.794930 05:45 DEBUG [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Starting 1 tasks in 8 threads 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group still has old name; still waiting for rename to take effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk.retries:rename_workspace_local_groups] Retrying: GroupRenameIncompleteError is allowed to retry (sleeping ~1s) 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group still has old name; still waiting for rename to take effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk.retries:rename_workspace_local_groups] Retrying: GroupRenameIncompleteError is allowed to retry (sleeping ~2s) 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group still has old name; still waiting for rename to take effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk.retries:rename_workspace_local_groups] Retrying: GroupRenameIncompleteError is allowed to retry (sleeping ~3s) 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename is assumed complete: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] waiting for renamed groups in the workspace 1/1, rps: 0.108/sec 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Finished 'waiting for renamed groups in the workspace' tasks: 0% results available (0/1). Took 0:00:09.241795 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900" < }, < { < "displayName": "role.labs.tika-ocr.maintain", < "id": "42451393964002" < }, < "... (60 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 60 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group enumeration reflects renamed group: 726451466196921 (renamed to rename-au9c-ucx-50JJ-ra78a5548f) 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900" < }, < { < "displayName": "role.labs.tika-ocr.maintain", < "id": "42451393964002" < }, < "... (60 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 60 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group enumeration reflects renamed group: 726451466196921 (renamed to rename-au9c-ucx-50JJ-ra78a5548f) 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group enumeration showed all 1 renamed groups; assuming complete. 05:45 INFO [databricks.labs.ucx:reflect_account_groups_on_workspace] UCX v0.41.1+120241010054235 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/reflect_account_groups_on_workspace.log 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing account groups with id,displayName,externalId... 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "role.github-emu.field-eng.labs-infra-write", < "externalId": "d8953329-3875-4ff2-bede-49cc584a1fbb", < "id": "14154893514742" < }, < "... (69 additional elements)" < ], < "itemsPerPage": 70, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 70 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 69 account groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta ... 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 57 Group 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ... 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups/448371539366696 < 200 OK < { < "displayName": "sdk-cdeA-ra78a5548f", < "groups": [], < "id": "448371539366696", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups/824705871265106 < 200 OK < { < "displayName": "sdk-jftY-ra78a5548f", < "groups": [], < "id": "824705871265106", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 3 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:reflect_account_groups_on_workspace] [hive_metastore.dummy_sicwj.groups] fetching groups inventory 05:45 DEBUG [databricks.labs.lsql.backends:reflect_account_groups_on_workspace] [spark][fetch] SELECT * FROM `hive_metastore`.`dummy_sicwj`.`groups` 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Starting to reflect 1 account groups into workspace for migration... 05:45 ERROR [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Skipping ucx-50JJ-ra78a5548f: group already exists in workspace 05:45 INFO [databricks.labs.ucx:apply_permissions] UCX v0.41.1+120241010054235 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/apply_permissions.log 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:apply_permissions] [hive_metastore.dummy_sicwj.groups] fetching groups inventory 05:45 DEBUG [databricks.labs.lsql.backends:apply_permissions] [spark][fetch] SELECT * FROM `hive_metastore`.`dummy_sicwj`.`groups` 05:45 INFO [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrating permissions for 1 account groups. 05:45 INFO [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrating permissions: rename-au9c-ucx-50JJ-ra78a5548f (workspace) -> ucx-50JJ-ra78a5548f (account) starting 05:45 DEBUG [databricks.sdk:apply_permissions] GET /api/2.0/preview/scim/v2/Me < 200 OK < { < "active": true, < "displayName": "labs-account-admin-identity", < "emails": [ < { < "primary": true, < "type": "work", < "value": "**REDACTED**" < } < ], < "externalId": "a0e683bc-a985-4535-8273-d54fed1cd03a", < "groups": [ < { < "$ref": "Groups/688239313962730", < "display": "labs.scope.account-admin", < "type": "direct", < "value": "**REDACTED**" < }, < "... (1 additional elements)" < ], < "id": "481119220561874", < "name": { < "givenName": "labs-account-admin-identity" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:User", < "... (1 additional elements)" < ], < "userName": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } 05:45 DEBUG [databricks.sdk:apply_permissions] POST /api/2.0/permissionmigration > { > "from_workspace_group_name": "rename-au9c-ucx-50JJ-ra78a5548f", > "size": 1000, > "to_account_group_name": "ucx-50JJ-ra78a5548f", > "workspace_id": 414838440637730 > } < 404 Not Found < { < "error_code": "NOT_FOUND", < "message": "Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace" < } 05:45 ERROR [databricks.labs.ucx.workspace_access.groups:apply_permissions] failed-group-migration: rename-au9c-ucx-50JJ-ra78a5548f -> ucx-50JJ-ra78a5548f: Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace 05:45 INFO [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrated 0 permissions for 0/1 groups successfully. 05:45 ERROR [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrating permissions failed for 1/1 groups. 05:45 ERROR [databricks.labs.ucx:apply_permissions] Execute `databricks workspace export //Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/apply_permissions.log` locally to troubleshoot with more details. Detected 1 failures: NotFound: Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace 05:45 DEBUG [databricks:apply_permissions] Task crash details Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/runtime.py", line 102, in trigger current_task(ctx) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/workflows.py", line 101, in apply_permissions elif migration_state.apply_to_renamed_groups(ctx.workspace_client): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/groups.py", line 92, in apply_to_renamed_groups return self._apply_to_groups(ws, renamed=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/groups.py", line 128, in _apply_to_groups raise ManyError(errors) databricks.labs.blueprint.parallel.ManyError: Detected 1 failures: NotFound: Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace 05:45 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ---------- 05:42 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/config.yml) doesn't exist. 05:42 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 05:42 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 05:42 INFO [databricks.labs.ucx.install] Fetching installations... 05:42 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 05:42 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:42 DEBUG [tests.integration.conftest] Waiting for clusters to start... 05:42 INFO [databricks.labs.ucx.install] Installing UCX v0.41.1+120241010054235 05:42 INFO [databricks.labs.ucx.install] Creating ucx schemas... 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 05:42 INFO [databricks.labs.ucx.install] Creating dashboards... 05:42 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment... 05:42 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views... 05:42 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration... 05:42 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV... 05:42 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main... 05:42 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates... 05:42 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive... 05:42 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups... 05:42 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main... 05:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 05:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 05:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental 05:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 05:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 05:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 05:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 05:42 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/README for the next steps. 05:42 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-groups-experimental job: https://DATABRICKS_HOST#job/1023920600021759 05:42 INFO [databricks.labs.ucx.installer.workflows] Started migrate-groups-experimental job: https://DATABRICKS_HOST#job/1023920600021759/runs/896928866636853 05:42 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of migrate-groups-experimental job: https://DATABRICKS_HOST#job/1023920600021759/runs/896928866636853 05:45 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS -------------- 05:45 INFO [databricks.labs.ucx:verify_metastore_attached] UCX v0.41.1+120241010054235 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/verify_metastore_attached.log 05:45 DEBUG [databricks.sdk:verify_metastore_attached] GET /api/2.1/unity-catalog/current-metastore-assignment < 200 OK < { < "TEST_SCHEMA_catalog_name": "hive_metastore", < "metastore_id": "8952c1e3-b265-4adf-98c3-6f755e2e1453", < "workspace_id": 414838440637730 < } 05:45 INFO [databricks.labs.ucx:rename_workspace_local_groups] UCX v0.41.1+120241010054235 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/rename_workspace_local_groups.log 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 57 Group 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/448371539366696 < 200 OK < { < "displayName": "sdk-cdeA-ra78a5548f", < "groups": [], < "id": "448371539366696", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/824705871265106 < 200 OK < { < "displayName": "sdk-jftY-ra78a5548f", < "groups": [], < "id": "824705871265106", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 3 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.dummy_sicwj.groups] fetching groups inventory 05:45 DEBUG [databricks.labs.lsql.backends:rename_workspace_local_groups] [spark][fetch] SELECT * FROM `hive_metastore`.`dummy_sicwj`.`groups` 05:45 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group ucx-50JJ-ra78a5548f defined in configuration does not exist on the groups table. Consider checking if the group exist in the workspace or re-running the assessment. 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.dummy_sicwj.groups] crawling new set of snapshot data for groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/448371539366696 < 200 OK < { < "displayName": "sdk-cdeA-ra78a5548f", < "groups": [], < "id": "448371539366696", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/824705871265106 < 200 OK < { < "displayName": "sdk-jftY-ra78a5548f", < "groups": [], < "id": "824705871265106", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 3 WorkspaceGroup 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing account groups with id,displayName,externalId... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "role.github-emu.field-eng.labs-infra-write", < "externalId": "d8953329-3875-4ff2-bede-49cc584a1fbb", < "id": "14154893514742" < }, < "... (69 additional elements)" < ], < "itemsPerPage": 70, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 70 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 69 account groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group listing provided, a subset of all groups will be migrated 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:rename_workspace_local_groups] [hive_metastore.dummy_sicwj.groups] found 1 new records for groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Starting to rename 1 groups for migration... 05:45 DEBUG [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Starting 1 tasks in 8 threads 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Renaming group: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] PATCH /api/2.0/preview/scim/v2/Groups/726451466196921 > { > "Operations": [ > { > "op": "replace", > "path": "displayName", > "value": "**REDACTED**" > } > ] > } < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] rename groups in the workspace 1/1, rps: 0.359/sec 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Finished 'rename groups in the workspace' tasks: 0% results available (0/1). Took 0:00:02.794930 05:45 DEBUG [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Starting 1 tasks in 8 threads 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group still has old name; still waiting for rename to take effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk.retries:rename_workspace_local_groups] Retrying: GroupRenameIncompleteError is allowed to retry (sleeping ~1s) 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group still has old name; still waiting for rename to take effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk.retries:rename_workspace_local_groups] Retrying: GroupRenameIncompleteError is allowed to retry (sleeping ~2s) 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 WARNING [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group still has old name; still waiting for rename to take effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk.retries:rename_workspace_local_groups] Retrying: GroupRenameIncompleteError is allowed to retry (sleeping ~3s) 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "rename-au9c-ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename has possibly taken effect: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group rename is assumed complete: ucx-50JJ-ra78a5548f (id=726451466196921) -> rename-au9c-ucx-50JJ-ra78a5548f 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] waiting for renamed groups in the workspace 1/1, rps: 0.108/sec 05:45 INFO [databricks.labs.blueprint.parallel:rename_workspace_local_groups] Finished 'waiting for renamed groups in the workspace' tasks: 0% results available (0/1). Took 0:00:09.241795 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900" < }, < { < "displayName": "role.labs.tika-ocr.maintain", < "id": "42451393964002" < }, < "... (60 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 60 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group enumeration reflects renamed group: 726451466196921 (renamed to rename-au9c-ucx-50JJ-ra78a5548f) 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName ... 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900" < }, < { < "displayName": "role.labs.tika-ocr.maintain", < "id": "42451393964002" < }, < "... (60 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:rename_workspace_local_groups] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Found 60 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group enumeration reflects renamed group: 726451466196921 (renamed to rename-au9c-ucx-50JJ-ra78a5548f) 05:45 DEBUG [databricks.labs.ucx.workspace_access.groups:rename_workspace_local_groups] Group enumeration showed all 1 renamed groups; assuming complete. 05:45 INFO [databricks.labs.ucx:reflect_account_groups_on_workspace] UCX v0.41.1+120241010054235 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/reflect_account_groups_on_workspace.log 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing account groups with id,displayName,externalId... 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/account/scim/v2/Groups?attributes=id,displayName,externalId < 200 OK < { < "Resources": [ < { < "displayName": "role.github-emu.field-eng.labs-infra-write", < "externalId": "d8953329-3875-4ff2-bede-49cc584a1fbb", < "id": "14154893514742" < }, < "... (69 additional elements)" < ], < "itemsPerPage": 70, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 70 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 69 account groups 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta ... 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "externalId": "8d2db608-4ed1-49f2-ad5c-fe942be7a4e1", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,externalId,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 57 Group 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ... 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=1&count=100 < 200 OK < { < "Resources": [ < { < "displayName": "role.labs.tempo.write", < "id": "22190446071900", < "meta": { < "resourceType": "Group" < } < }, < "... (61 additional elements)" < ], < "itemsPerPage": 62, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 1, < "totalResults": 62 < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups/448371539366696 < 200 OK < { < "displayName": "sdk-cdeA-ra78a5548f", < "groups": [], < "id": "448371539366696", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups/726451466196921 < 200 OK < { < "displayName": "ucx-50JJ-ra78a5548f", < "entitlements": [ < { < "value": "**REDACTED**" < } < ], < "groups": [], < "id": "726451466196921", < "members": [ < { < "$ref": "Users/6996727931128055", < "display": "dummy-rn7g-ra78a5548f@example.com", < "value": "**REDACTED**" < } < ], < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups/824705871265106 < 200 OK < { < "displayName": "sdk-jftY-ra78a5548f", < "groups": [], < "id": "824705871265106", < "meta": { < "resourceType": "WorkspaceGroup" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:Group" < ] < } 05:45 DEBUG [databricks.sdk:reflect_account_groups_on_workspace] GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta&startIndex=63&count=100 < 200 OK < { < "itemsPerPage": 0, < "schemas": [ < "urn:ietf:params:scim:api:messages:2.0:ListResponse" < ], < "startIndex": 63, < "totalResults": 62 < } 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Found 3 WorkspaceGroup 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:reflect_account_groups_on_workspace] [hive_metastore.dummy_sicwj.groups] fetching groups inventory 05:45 DEBUG [databricks.labs.lsql.backends:reflect_account_groups_on_workspace] [spark][fetch] SELECT * FROM `hive_metastore`.`dummy_sicwj`.`groups` 05:45 INFO [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Starting to reflect 1 account groups into workspace for migration... 05:45 ERROR [databricks.labs.ucx.workspace_access.groups:reflect_account_groups_on_workspace] Skipping ucx-50JJ-ra78a5548f: group already exists in workspace 05:45 INFO [databricks.labs.ucx:apply_permissions] UCX v0.41.1+120241010054235 After job finishes, see debug logs at /Workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/apply_permissions.log 05:45 DEBUG [databricks.labs.ucx.framework.crawlers:apply_permissions] [hive_metastore.dummy_sicwj.groups] fetching groups inventory 05:45 DEBUG [databricks.labs.lsql.backends:apply_permissions] [spark][fetch] SELECT * FROM `hive_metastore`.`dummy_sicwj`.`groups` 05:45 INFO [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrating permissions for 1 account groups. 05:45 INFO [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrating permissions: rename-au9c-ucx-50JJ-ra78a5548f (workspace) -> ucx-50JJ-ra78a5548f (account) starting 05:45 DEBUG [databricks.sdk:apply_permissions] GET /api/2.0/preview/scim/v2/Me < 200 OK < { < "active": true, < "displayName": "labs-account-admin-identity", < "emails": [ < { < "primary": true, < "type": "work", < "value": "**REDACTED**" < } < ], < "externalId": "a0e683bc-a985-4535-8273-d54fed1cd03a", < "groups": [ < { < "$ref": "Groups/688239313962730", < "display": "labs.scope.account-admin", < "type": "direct", < "value": "**REDACTED**" < }, < "... (1 additional elements)" < ], < "id": "481119220561874", < "name": { < "givenName": "labs-account-admin-identity" < }, < "schemas": [ < "urn:ietf:params:scim:schemas:core:2.0:User", < "... (1 additional elements)" < ], < "userName": "0a330eb5-dd51-4d97-b6e4-c474356b1d5d" < } 05:45 DEBUG [databricks.sdk:apply_permissions] POST /api/2.0/permissionmigration > { > "from_workspace_group_name": "rename-au9c-ucx-50JJ-ra78a5548f", > "size": 1000, > "to_account_group_name": "ucx-50JJ-ra78a5548f", > "workspace_id": 414838440637730 > } < 404 Not Found < { < "error_code": "NOT_FOUND", < "message": "Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace" < } 05:45 ERROR [databricks.labs.ucx.workspace_access.groups:apply_permissions] failed-group-migration: rename-au9c-ucx-50JJ-ra78a5548f -> ucx-50JJ-ra78a5548f: Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace 05:45 INFO [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrated 0 permissions for 0/1 groups successfully. 05:45 ERROR [databricks.labs.ucx.workspace_access.groups:apply_permissions] Migrating permissions failed for 1/1 groups. 05:45 ERROR [databricks.labs.ucx:apply_permissions] Execute `databricks workspace export //Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.au9c/logs/migrate-groups-experimental/run-896928866636853-0/apply_permissions.log` locally to troubleshoot with more details. Detected 1 failures: NotFound: Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace 05:45 DEBUG [databricks:apply_permissions] Task crash details Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/runtime.py", line 102, in trigger current_task(ctx) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/workflows.py", line 101, in apply_permissions elif migration_state.apply_to_renamed_groups(ctx.workspace_client): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/groups.py", line 92, in apply_to_renamed_groups return self._apply_to_groups(ws, renamed=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fcb14537-6230-435c-82f3-74607cb0af30/lib/python3.11/site-packages/databricks/labs/ucx/workspace_access/groups.py", line 128, in _apply_to_groups raise ManyError(errors) databricks.labs.blueprint.parallel.ManyError: Detected 1 failures: NotFound: Account group ucx-50JJ-ra78a5548f does not exist or is not assigned to workspace 05:45 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ---------- 05:45 INFO [databricks.labs.ucx.install] Deleting UCX v0.41.1+120241010054235 from https://DATABRICKS_HOST 05:45 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_sicwj 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=16885909085612, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=747684048954831, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1077130456420570, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=834897965761865, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=931857278718508, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=131734783560505, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=964073826464826, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=254871791865934, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=97082283761263, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=175603697539382, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1023920600021759, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=89674569114375, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=633415622531094, as it is no longer needed 05:45 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=243699560967493, as it is no longer needed 05:45 INFO [databricks.labs.ucx.install] Deleting cluster policy 05:45 INFO [databricks.labs.ucx.install] Deleting secret scope 05:45 INFO [databricks.labs.ucx.install] UnInstalling UCX complete [gw0] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python ```

_{Running from nightly #225}

databrickslabs / ucx

Test failure: `test_experimental_permissions_migration_for_group_with_same_name` #2903