search

databrickslabs / ucx

Automated migrations to Unity Catalog
Other
231 stars 79 forks source link

Test failure: `test_running_real_validate_groups_permissions_job_fails` #1143

Closed github-actions[bot] closed 7 months ago

github-actions[bot] commented 7 months ago
❌ test_running_real_validate_groups_permissions_job_fails: databricks.sdk.errors.platform.Unknown: validate_groups_permissions: ValueError: Couldn't find permission for object type cluster-policies with id 000CB9EA5FF4ACFB (2m19.096s) ``` databricks.sdk.errors.platform.Unknown: validate_groups_permissions: ValueError: Couldn't find permission for object type cluster-policies with id 000CB9EA5FF4ACFB acl to be applied=[AccessControlRequest(group_name=None, permission_level=, service_principal_name='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', user_name=None), AccessControlRequest(group_name='sdk-wlF9', permission_level=, service_principal_name=None, user_name=None), AccessControlRequest(group_name='admins', permission_level=, service_principal_name=None, user_name=None)] acl found in the object=[AccessControlRequest(group_name='admins', permission_level=, service_principal_name=None, user_name=None)] 23:41 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sbmf6: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sbmf6 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sbmf6', metastore_id=None, name='ucx_sbmf6', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw0] linux -- Python 3.10.13 /home/runner/work/ucx/ucx/.venv/bin/python 23:41 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sbmf6: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sbmf6 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sbmf6', metastore_id=None, name='ucx_sbmf6', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 23:41 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-wlF9: https://DATABRICKS_HOST#setting/accounts/groups/784771829979677 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-wlF9', entitlements=[], external_id=None, groups=[], id='784771829979677', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 23:41 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000CB9EA5FF4ACFB 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000CB9EA5FF4ACFB') 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000CB9EA5FF4ACFB [group_name admins CAN_USE] -> [group_name sdk-wlF9 CAN_USE] 23:41 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.llEb/config.yml) doesn't exist. 23:41 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 23:41 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 23:41 INFO [databricks.labs.ucx.install] Fetching installations... 23:41 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 23:41 INFO [databricks.labs.ucx.install] Installing UCX v0.19.1+2020240327234114 23:41 INFO [databricks.labs.ucx.install] Creating dashboards... 23:41 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views... 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment... 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates... 23:41 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Estimates)... 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 01_0_group_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 01_0_group_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_metastore_assignment.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_metastore_assignment.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_data_modeling.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_data_modeling.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 03_0_data_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 03_0_data_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main... 23:41 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Main)... 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV... 23:42 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Azure)... 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive... 23:42 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Interactive)... 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_interactive.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_interactive.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_cluster_summary.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_cluster_summary.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.installer.workflows] Creating jobs from tasks in main 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=099-destroy-schema 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 23:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 23:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 23:42 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.llEb/README for the next steps. 23:42 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions 23:42 INFO [databricks.labs.ucx.workspace_access.generic] Listed cluster-policies in 0:00:00.100453 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 14 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 14 23:42 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sbmf6.permissions] found 14 new records for permissions 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Saved 14 to hive_metastore.ucx_sbmf6.permissions 23:42 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/615790071516542 23:41 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sbmf6: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sbmf6 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sbmf6', metastore_id=None, name='ucx_sbmf6', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 23:41 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-wlF9: https://DATABRICKS_HOST#setting/accounts/groups/784771829979677 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-wlF9', entitlements=[], external_id=None, groups=[], id='784771829979677', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 23:41 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000CB9EA5FF4ACFB 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000CB9EA5FF4ACFB') 23:41 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000CB9EA5FF4ACFB [group_name admins CAN_USE] -> [group_name sdk-wlF9 CAN_USE] 23:41 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.llEb/config.yml) doesn't exist. 23:41 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 23:41 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 23:41 INFO [databricks.labs.ucx.install] Fetching installations... 23:41 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 23:41 INFO [databricks.labs.ucx.install] Installing UCX v0.19.1+2020240327234114 23:41 INFO [databricks.labs.ucx.install] Creating dashboards... 23:41 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views... 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment... 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates... 23:41 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Estimates)... 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 01_0_group_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 01_0_group_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_metastore_assignment.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_metastore_assignment.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_data_modeling.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_data_modeling.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 03_0_data_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 03_0_data_migration.md because it's a text widget 23:41 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main... 23:41 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Main)... 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV... 23:42 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Azure)... 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive... 23:42 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [LLEB] UCX Assessment (Interactive)... 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_interactive.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_interactive.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_cluster_summary.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_cluster_summary.md because it's a text widget 23:42 DEBUG [databricks.labs.ucx.installer.workflows] Creating jobs from tasks in main 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=099-destroy-schema 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 23:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 23:42 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 23:42 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 23:42 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.llEb/README for the next steps. 23:42 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions 23:42 INFO [databricks.labs.ucx.workspace_access.generic] Listed cluster-policies in 0:00:00.100453 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 14 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 14 23:42 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sbmf6.permissions] found 14 new records for permissions 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table 23:42 INFO [databricks.labs.ucx.workspace_access.manager] Saved 14 to hive_metastore.ucx_sbmf6.permissions 23:42 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/615790071516542 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000CB9EA5FF4ACFB [group_name admins CAN_USE] -> [group_name sdk-wlF9 CAN_USE] 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000CB9EA5FF4ACFB') 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='sdk-wlF9', entitlements=[], external_id=None, groups=[], id='784771829979677', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 23:43 INFO [databricks.labs.ucx.install] Deleting UCX v0.19.1+2020240327234308 from https://DATABRICKS_HOST 23:43 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sbmf6 23:43 INFO [databricks.labs.ucx.install] Deleting jobs 23:43 INFO [databricks.labs.ucx.install] Deleting 099-destroy-schema job_id=47663935536023. 23:43 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=895452399371860. 23:43 INFO [databricks.labs.ucx.install] Deleting assessment job_id=334060546277715. 23:43 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=65348978884405. 23:43 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=615790071516542. 23:43 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=770773474527054. 23:43 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=1081323959608005. 23:43 INFO [databricks.labs.ucx.install] Deleting cluster policy 23:43 INFO [databricks.labs.ucx.install] Deleting secret scope 23:43 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 23:43 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sbmf6', metastore_id=None, name='ucx_sbmf6', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw0] linux -- Python 3.10.13 /home/runner/work/ucx/ucx/.venv/bin/python ```

Running from nightly #1

github-actions[bot] commented 7 months ago
❌ test_running_real_validate_groups_permissions_job_fails: databricks.sdk.errors.platform.Unknown: validate_groups_permissions: ValueError: Couldn't find permission for object type cluster-policies with id 0008AD7828BCE4E4 (1m38.83s) ``` databricks.sdk.errors.platform.Unknown: validate_groups_permissions: ValueError: Couldn't find permission for object type cluster-policies with id 0008AD7828BCE4E4 acl to be applied=[AccessControlRequest(group_name=None, permission_level=, service_principal_name='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', user_name=None), AccessControlRequest(group_name='sdk-0cjV', permission_level=, service_principal_name=None, user_name=None), AccessControlRequest(group_name='admins', permission_level=, service_principal_name=None, user_name=None)] acl found in the object=[AccessControlRequest(group_name='admins', permission_level=, service_principal_name=None, user_name=None)] 07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s3bkr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s3bkr 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3bkr', metastore_id=None, name='ucx_s3bkr', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw5] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python 07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s3bkr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s3bkr 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3bkr', metastore_id=None, name='ucx_s3bkr', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 07:03 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-0cjV: https://DATABRICKS_HOST#setting/accounts/groups/632652426937708 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-0cjV', entitlements=[], external_id=None, groups=[], id='632652426937708', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 07:03 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0008AD7828BCE4E4 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0008AD7828BCE4E4') 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0008AD7828BCE4E4 [group_name admins CAN_USE] -> [group_name sdk-0cjV CAN_USE] 07:03 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Tl8Y/config.yml) doesn't exist. 07:03 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 07:03 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 07:03 INFO [databricks.labs.ucx.install] Fetching installations... 07:03 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 07:03 INFO [databricks.labs.ucx.install] Installing UCX v0.20.1+220240329070354 07:03 INFO [databricks.labs.ucx.install] Creating dashboards... 07:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 07:03 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views... 07:03 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment... 07:03 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates... 07:03 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Estimates)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 01_0_group_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 01_0_group_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_metastore_assignment.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_metastore_assignment.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_data_modeling.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_data_modeling.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 03_0_data_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 03_0_data_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main... 07:04 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Main)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 10___data_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 10___data_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 15___storage_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 15___storage_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 40___last_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 40___last_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 20___compute_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 20___compute_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00___assessment_overview.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00___assessment_overview.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 05___findings_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 05___findings_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 30_0_job_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 30_0_job_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV... 07:04 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Azure)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive... 07:04 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Interactive)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_interactive.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_interactive.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_cluster_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_cluster_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.installer.workflows] Creating jobs from tasks in main 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 07:04 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 07:04 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=099-destroy-schema 07:04 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Tl8Y/README for the next steps. 07:04 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions 07:04 INFO [databricks.labs.ucx.workspace_access.generic] Listed cluster-policies in 0:00:00.036419 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 27 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 27 07:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3bkr.permissions] found 27 new records for permissions 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Saved 27 to hive_metastore.ucx_s3bkr.permissions 07:04 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/236689369099666 07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s3bkr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s3bkr 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3bkr', metastore_id=None, name='ucx_s3bkr', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) 07:03 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group sdk-0cjV: https://DATABRICKS_HOST#setting/accounts/groups/632652426937708 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='sdk-0cjV', entitlements=[], external_id=None, groups=[], id='632652426937708', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 07:03 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0008AD7828BCE4E4 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0008AD7828BCE4E4') 07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0008AD7828BCE4E4 [group_name admins CAN_USE] -> [group_name sdk-0cjV CAN_USE] 07:03 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Tl8Y/config.yml) doesn't exist. 07:03 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration 07:03 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data. 07:03 INFO [databricks.labs.ucx.install] Fetching installations... 07:03 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy. 07:03 INFO [databricks.labs.ucx.install] Installing UCX v0.20.1+220240329070354 07:03 INFO [databricks.labs.ucx.install] Creating dashboards... 07:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 07:03 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views... 07:03 DEBUG [databricks.labs.ucx.framework.dashboards] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment... 07:03 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates... 07:03 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Estimates)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 01_0_group_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 01_0_group_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_metastore_assignment.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_metastore_assignment.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_data_modeling.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_data_modeling.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 03_0_data_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 03_0_data_migration.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main... 07:04 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Main)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 10___data_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 10___data_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 15___storage_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 15___storage_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 40___last_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 40___last_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 20___compute_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 20___compute_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00___assessment_overview.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00___assessment_overview.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 05___findings_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 05___findings_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 30_0_job_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 30_0_job_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV... 07:04 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Azure)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Reading dashboard folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive... 07:04 INFO [databricks.labs.ucx.framework.dashboards] Creating dashboard [TL8Y] UCX Assessment (Interactive)... 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 00_0_interactive.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 00_0_interactive.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping query 02_0_cluster_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.framework.dashboards] Skipping viz 02_0_cluster_summary.md because it's a text widget 07:04 DEBUG [databricks.labs.ucx.installer.workflows] Creating jobs from tasks in main 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups 07:04 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 07:04 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment 07:04 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=099-destroy-schema 07:04 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Tl8Y/README for the next steps. 07:04 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions 07:04 INFO [databricks.labs.ucx.workspace_access.generic] Listed cluster-policies in 0:00:00.036419 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 27 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 27 07:04 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s3bkr.permissions] found 27 new records for permissions 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table 07:04 INFO [databricks.labs.ucx.workspace_access.manager] Saved 27 to hive_metastore.ucx_s3bkr.permissions 07:04 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/236689369099666 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 0008AD7828BCE4E4 [group_name admins CAN_USE] -> [group_name sdk-0cjV CAN_USE] 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='0008AD7828BCE4E4') 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='sdk-0cjV', entitlements=[], external_id=None, groups=[], id='632652426937708', members=[], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[]) 07:05 INFO [databricks.labs.ucx.install] Deleting UCX v0.20.1+220240329070524 from https://DATABRICKS_HOST 07:05 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s3bkr 07:05 INFO [databricks.labs.ucx.install] Deleting jobs 07:05 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=544554538587612. 07:05 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=182479142339217. 07:05 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1089142245849955. 07:05 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=236689369099666. 07:05 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=611736584486607. 07:05 INFO [databricks.labs.ucx.install] Deleting assessment job_id=744997758976671. 07:05 INFO [databricks.labs.ucx.install] Deleting 099-destroy-schema job_id=451193489856415. 07:05 INFO [databricks.labs.ucx.install] Deleting cluster policy 07:05 INFO [databricks.labs.ucx.install] Deleting secret scope 07:05 INFO [databricks.labs.ucx.install] UnInstalling UCX complete 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures 07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3bkr', metastore_id=None, name='ucx_s3bkr', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None) [gw5] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python ```

Running from nightly #3