Azure ADF using Azure Batch throws Shared Access Signature generation error with Managed Identity

DeepMalh44 commented 3 years ago

I am working on a simple Azure Data Factory pipeline where I have simply added a Batch Service and in that specified the Batch Service account (which I have created thru linked service and tested the connection is working). In the command I am just running a simple "ls" command and when I do a debug run I get this error: "Cannot create Shared Access Signature unless Account Key credentials are used." I have following linked services "Azure Batch", "Azure Blob Storage" and Key Vault (where we store the access key). All linked services connections are working properly. For Azure Blog Storage we are using "Managed Identity" and not "Account Key authentication) and I am still not sure why the above error with Managed identity used. Is this a bug where even after selecting the Authentication Type as MANAGED IDENTITY the storage account access us going thru Account Key access?

Any help on how to fix this error: "Cannot create Shared Access Signature unless Account Key credentials are used."

Azure Batch Linked service:

Azure Storage Linked service:

Azure Data factory pipeline:

kesharma commented 3 years ago

@ketaanhshah this is currently a bug in custom activity unable to support storage LS set up with managed identity and only support account key. This is expected to be fixed by the end of next month. Edit: Batch service recently added support for storage with MI required to available to support this feature from ADF. The approximate ETA for the support in ADF would be expected by the end of Sept.

DeepMalh44 commented 2 years ago

@kesharma Hello sorry for getting back on this thread again, was the above issue fixed last year of Sept 2021? As one of our customers is facing same issue now with the Managed identity ... can you please provide some update? Thank you

AllainPL commented 1 year ago

After a rather lengthy deep dive I got it to work. Not sure if you have the exact same scenario but I will explain mine, maybe it will be useful for you.

ADF with linked services - Azure Batch and Azure Blob storage

What you need to start is a User assigned managed identity and you need to set it ALMOST everywhere possible.

Storage account

Add the managed identity as Storage Blob Data Owner for your storage account

Batch account

In batch account under Setttings->Identity, select user-assigned and add the one you have.

In batch account under Setttings->Storage account switch authentication mode to Batch account managed identity In batch account under Setttings->Storage account add the identity under the Node identity reference

In batch account under Features->Pools when creating a pool switch to User-assigned identity and this was critical as on this part I was stuck for a long time.

Data factory

On the data factory (in azure portal not studio) under Settings-> Managed identities enable System assigned

Batch account

Go back to the batch account and add the data factory system identity as a contributor (not sure how strong role is needed)

Data factory studio

Under Manage->Security->Credentials Add your user assigned managed identity

Under Manage->Connections->Linked services

for the Azure batch connection set it to system assigned managed identity

for the Storage account select user assigned managed identity

Now 🤞and hope I did put all the needed details :D

If everything works fine live long and prosper 🖖 and be safe and happy without ugly account keys :)

Note on Identity in batch pool

As far as I checked there is no way to changed it for an already existing pool via UI. Neither New-AzBatchPool or Set-AzBatchPool can do it. So, the only way is via arm templates or using REST. Sample below:

$identityPatch = '{
    "identity": {
        "type": "UserAssigned",
        "userAssignedIdentities": {
            "/subscriptions/__subscriptionID__/resourceGroups/__resourceGroupWithManagedIdentity__/providers/Microsoft.ManagedIdentity/userAssignedIdentities/__userAssignedManageIdentity__": {}
        }
    }
}'
$patchParams = @{
      ResourceGroupName = '__resourceGroupName__'
      ResourceProviderName = 'Microsoft.Batch'
      ResourceType = 'batchAccounts'
      Name = '__batchAccountName__/pools/__poolName__'
      ApiVersion = '2022-10-01'
      Payload = $identityString
      Method = 'PATCH'
    }
Invoke-AzRestMethod @patchParams

Documentations that got me unstuck https://learn.microsoft.com/en-us/troubleshoot/azure/general/azure-batch-node-unusable-state https://learn.microsoft.com/en-us/troubleshoot/azure/general/use-managed-identities-azure-batch-account-pool esspecialy this table

Azure / Azure-DataFactory