search

databricks / cli

Databricks CLI
Other
124 stars 47 forks source link

[Azure] Unity Catalog: Perform permission check during Storage Credential creation instead of role ID lookup #1486

Open mixam24 opened 2 months ago

mixam24 commented 2 months ago

Describe the issue

I'm trying to create a new storage credential in Unity Catalog using my personal user that has Unity Metastore Admin permissions. Due to security reasons, our organization does not allow personal users to have the built-in "Contributor" role. Instead, we assign a custom one, which is almost identical to the built-in role. Hence, my personal identity does not have built-in "Contributor" role.

The custom role definition is as follows:

{
  "assignableScopes": [
    "/subscriptions/6e85b02d-ec23-493e-bf2a-4be65492e6b5",
    "/subscriptions/96875dfe-c52a-4851-83b0-254863be206d",
    "/providers/Microsoft.Management/managementgroups/2d6bfa44-b523-4579-a70d-a145eaf272d5"
  ],
  "createdBy": "70b08e41-e31b-43db-8c57-b2ed5681d82c",
  "createdOn": "2019-07-24T12:59:19.599048+00:00",
  "description": "[Version 1.2] Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.",
  "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "name": "ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "permissions": [
    {
      "actions": ["*"],
      "condition": null,
      "conditionVersion": null,
      "dataActions": [],
      "notActions": [
        "Microsoft.Authorization/*/Delete",
        "Microsoft.Authorization/*/Write",
        "Microsoft.Authorization/elevateAccess/Action",
        "Microsoft.Blueprint/blueprintAssignments/write",
        "Microsoft.Blueprint/blueprintAssignments/delete",
        "Microsoft.Compute/galleries/share/action",
        "Microsoft.Purview/consents/write",
        "Microsoft.Purview/consents/delete",
        "Microsoft.Subscription/rename/action",
        "Microsoft.Resources/deploymentStacks/manageDenySetting/action"
      ],
      "notDataActions": []
    }
  ],
  "roleName": "Custom Contributor",
  "roleType": "CustomRole",
  "type": "Microsoft.Authorization/roleDefinitions",
  "updatedBy": "366d6245-093c-4bc0-b8a0-dce37cdf383e",
  "updatedOn": "2024-05-16T08:53:05.530192+00:00"
}

The attempt to create a storage credential fails with the following error: Creation of a storage credential required the contributor role over the corresponding access connector with ID '...'. Please contact your account admin.

I believe that the service performs role assignment lookups only by ID and does not check that my user has the necessary permissions to perform the action.

Steps to reproduce the behavior

  1. List the identity role assignments over the databricks access connector

    ~/ ❯ az role assignment list --assignee c0409af9-7ccb-4e18-adf2-93dd50a39adc \
    --include-groups --include-inherited  \
    --scope "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog"

[
{
  "condition": null,
  "conditionVersion": null,
  "createdBy": "51d2f4f3-b46e-4c80-a394-0fcd87449619",
  "createdOn": "2023-09-23T07:49:44.104274+00:00",
  "delegatedManagedIdentityResourceId": null,
  "description": null,
  "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleAssignments/441d2705-cbb4-40af-ad30-31d0f3422436",
  "name": "441d2705-cbb4-40af-ad30-31d0f3422436",
  "principalId": "8bc9963a-1e72-4314-8703-3cd680368384",
  "principalName": "My Subscription Group",
  "principalType": "Group",
  "roleDefinitionId": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7",
  "roleDefinitionName": "Reader",
  "scope": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f",
  "type": "Microsoft.Authorization/roleAssignments",
  "updatedBy": "51d2f4f3-b46e-4c80-a394-0fcd87449619",
  "updatedOn": "2023-09-23T07:49:44.104274+00:00"
},
{
  "condition": null,
  "conditionVersion": null,
  "createdBy": "dde92205-34c0-4028-ba44-b2cd3d096701",
  "createdOn": "2024-03-20T12:55:28.371679+00:00",
  "delegatedManagedIdentityResourceId": null,
  "description": null,
  "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleAssignments/82daa968-312f-42e7-8934-7c596545607b",
  "name": "82daa968-312f-42e7-8934-7c596545607b",
  "principalId": "8bc9963a-1e72-4314-8703-3cd680368384",
  "principalName": "My Subscription Group",
  "principalType": "Group",
  "roleDefinitionId": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "roleDefinitionName": "Custom Contributor",
  "scope": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f",
  "type": "Microsoft.Authorization/roleAssignments",
  "updatedBy": "dde92205-34c0-4028-ba44-b2cd3d096701",
  "updatedOn": "2024-03-13T11:55:33.378901+00:00"
}
]

  2. Describe the Custom Contributor role definition

    ~/ ❯ az role definition list --name "Custom Contributor"

[
{
  "assignableScopes": [
    "/subscriptions/6e85b02d-ec23-493e-bf2a-4be65492e6b5",
    "/subscriptions/96875dfe-c52a-4851-83b0-254863be206d",
    "/providers/Microsoft.Management/managementgroups/2d6bfa44-b523-4579-a70d-a145eaf272d5"
  ],
  "createdBy": "70b08e41-e31b-43db-8c57-b2ed5681d82c",
  "createdOn": "2019-07-24T12:59:19.599048+00:00",
  "description": "[Version 1.2] Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.",
  "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "name": "ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "permissions": [
    {
      "actions": [
        "*"
      ],
      "condition": null,
      "conditionVersion": null,
      "dataActions": [],
      "notActions": [
        "Microsoft.Authorization/*/Delete",
        "Microsoft.Authorization/*/Write",
        "Microsoft.Authorization/elevateAccess/Action",
        "Microsoft.Blueprint/blueprintAssignments/write",
        "Microsoft.Blueprint/blueprintAssignments/delete",
        "Microsoft.Compute/galleries/share/action",
        "Microsoft.Purview/consents/write",
        "Microsoft.Purview/consents/delete",
        "Microsoft.Subscription/rename/action",
        "Microsoft.Resources/deploymentStacks/manageDenySetting/action"
      ],
      "notDataActions": []
    }
  ],
  "roleName": "Custom Contributor",
  "roleType": "CustomRole",
  "type": "Microsoft.Authorization/roleDefinitions",
  "updatedBy": "366d6245-093c-4bc0-b8a0-dce37cdf383e",
  "updatedOn": "2024-05-16T08:53:05.530192+00:00"
}
]
  3. Obtain AAD access token and set databricks host 
    ~/ ❯ export DATABRICKS_HOST="https://adb-1234567880123456.7.azuredatabricks.net"
~/ ❯ export DATABRICKS_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq .accessToken -r)

  4. Demonstrate that the identity used has Metastore Admin permissions

    ~/ ❯ databricks unity-catalog storage-credentials list

{
"storage_credentials": [
  {
    "name": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2-storage-credential-1234567890123",
    "azure_managed_identity": {
      "access_connector_id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/databricks-unity-metastore/providers/Microsoft.Databricks/accessConnectors/unity-metastore-westeu",
      "managed_identity_id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/databricks-unity-metastore/providers/Microsoft.ManagedIdentity/userAssignedIdentities/unity-metastore-westeu",
      "credential_id": "c8aab434-60bc-44c0-a00a-84cf95b5c23c"
    },
    "read_only": false,
    "owner": "admin@example.com",
    "id": "21ce434a-78c1-4a44-9850-c6ff2cceea6f",
    "metastore_id": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2",
    "created_at": 1696498690290,
    "created_by": "admin@example.com",
    "updated_at": 1696498690290,
    "updated_by": "admin@example.com",
    "used_for_managed_storage": true,
    "full_name": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2-storage-credential-1234567890123",
    "securable_type": "STORAGE_CREDENTIAL",
    "securable_kind": "STORAGE_CREDENTIAL_AZURE_MI",
    "isolation_mode": "ISOLATION_MODE_OPEN"
  },
  ...
]
}

  5. Create storage credential using access connector ID and managed identity ID

    ~/ ❯ databricks unity-catalog storage-credentials create --name test \
    --az-mi-access-connector-id "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog" \
    --az-mi-id "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my-connector-unity-catalog-mi"

    Traceback (most recent call last):
    File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py", line 166, in perform_query
      resp.raise_for_status()
    File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
      raise HTTPError(http_error_msg, response=self)
  requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://adb-1234567880123456.7.azuredatabricks.net/api/2.1/unity-catalog/storage-credentials

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/bin/databricks", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
      return __callback(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/cred_cli.py", line 114, in wrapper
      f(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/databricks_cli/configure/config.py", line 67, in decorator
      return function(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/cred_cli.py", line 166, in create_credential_cli
      cred_json = UnityCatalogApi(api_client).create_storage_credential(data, skip_val)
    File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/api.py", line 103, in create_storage_credential
      return self.client.create_storage_credential(cred_spec, skip_validation)
    File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/uc_service.py", line 175, in create_storage_credential
      return self.client.perform_query('POST', url, data=cred_spec, headers=headers)
    File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py", line 174, in perform_query
      raise requests.exceptions.HTTPError(message, response=e.response)
  requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://adb-1234567880123456.7.azuredatabricks.net/api/2.1/unity-catalog/storage-credentials
   Response from server:
   { 'details': [ { '@type': 'type.googleapis.com/google.rpc.ErrorInfo',
                   'domain': 'unity-catalog.databricks.com',
                   'metadata': { 'accessConnectorId': '/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog'},
                   'reason': 'UC_STORAGE_CREDENTIAL_INVALID_CLOUD_PERMISSIONS'},
                 { '@type': 'type.googleapis.com/google.rpc.RequestInfo',
                   'request_id': 'eb684711-61ca-441b-ad34-8acc1e632b48',
                   'serving_data': ''}],
    'error_code': 'PERMISSION_DENIED',
    'message': 'Creating a storage credential requires the contributor role over '
               'the corresponding access connector with ID '
               '/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog. '
               'Please contact your account admin.'}

Expected Behavior

Creation of the storage credential succeeded since the identity used has enough access permissions.

Actual Behavior

The attempt of a storage credential creation is failing with the following error: Creation of a storage credential required the contributor role over the corresponding access connector with ID '...'. Please contact your account admin.

OS and CLI version

~/ ❯ uname -s
Darwin

~/ ❯ echo $OSTYPE
darwin23.0

~/ ❯ databricks --version

Version 0.17.4

Is this a regression?

n.a.

Debug Logs

n.a.

mixam24 commented 1 month ago

Any comments on this one?

pietern commented 1 month ago

@mixam24 This sounds like a product feature/issue, not a CLI issue. I forwarded the question internally.

I doubt that it matters, but could you try doing the same with the new CLI? You're using the legacy Python one.

Installation instructions can be found at https://docs.databricks.com/en/dev-tools/cli/install.html.

mixam24 commented 1 month ago

@pietern thanks for your reply.

This sounds like a product feature/issue, not a CLI issue.

I agree, the same problem arises in UI and in terraform (= anything that interacts with REST API).

I forwarded the question internally.

Could you kindly recommend an alternative communication channel for the discussion of this issue? I have already contacted Microsoft support, but I have serious doubts regarding their proficiency in the topic and overall effectiveness of the communication.