Unable to run deployment script attached to storage account with AllowSharedKeyAccess false

[speters82]

Bicep version

0.28.1

Describe the bug

Recent security initiatives at my company are forbidding the use of storage account keys in favor of MSI. However, when I try to use MSI to authenticate my deployment script to my storage account and I create the storage account with allowSharedKeyAccess set to false, it just spins there for a while (about 10-15 minutes) before failing.

To Reproduce Steps to reproduce the behavior:

1. Put the bicep below into a file named scripttest.bicep
2. Log into your Azure Subscription on the Az CLI.
3. Create the resource group

$ az group create --name rg_scripttest --location westus

4. Deploy the bicep file

$ az deployment group create --resource-group rg_scripttest --template-file ./scripttest.bicep --parameters prefix=test1

Expected: The deployment succeeds. Actual: The deployment spins for a while before failing

@maxLength(10) // Required maximum length, because the storage account has a maximum of 26 characters
param prefix string
param location string = resourceGroup().location
param userAssignedIdentityName string = '${prefix}Identity'
param storageAccountName string = '${prefix}stg${uniqueString(resourceGroup().id)}'
param vnetName string = '${prefix}Vnet'
param subnetName string = '${prefix}Subnet'
param utcValue string = utcNow()

resource vnet 'Microsoft.Network/virtualNetworks@2023-05-01' = {
  name: vnetName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    enableDdosProtection: false
    subnets: [
      {
        name: subnetName
        properties: {
          addressPrefix: '10.0.0.0/24'
          serviceEndpoints: [
            {
              service: 'Microsoft.Storage'
            }
          ]
          delegations: [
            {
              name: 'Microsoft.ContainerInstance.containerGroups'
              properties: {
                serviceName: 'Microsoft.ContainerInstance/containerGroups'
              }
            }
          ]
        }
      }
    ]
  }
}

resource subnet 'Microsoft.Network/virtualNetworks/subnets@2023-05-01' existing = {
  parent: vnet
  name: subnetName
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: storageAccountName
  location: location
  sku: {
    name: 'Standard_LRS'
  }
  kind: 'StorageV2'
  properties: {
    allowBlobPublicAccess: false
    minimumTlsVersion: 'TLS1_2'
    allowSharedKeyAccess: false
    networkAcls: {
      bypass: 'AzureServices'
      virtualNetworkRules: [
        {
          id: subnet.id
          action: 'Allow'
          state: 'Succeeded'
        }
      ]
      defaultAction: 'Deny'
    }
  }
}

resource userAssignedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: userAssignedIdentityName
  location: location
}

resource storageFileDataPrivilegedContributor 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: '69566ab7-960f-475b-8e7c-b3118f30c6bd' // Storage File Data Privileged Contributor
  scope: tenant()
}

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  scope: storageAccount

  name: guid(storageFileDataPrivilegedContributor.id, userAssignedIdentity.id, storageAccount.id)
  properties: {
    principalId: userAssignedIdentity.properties.principalId
    roleDefinitionId: storageFileDataPrivilegedContributor.id
    principalType: 'ServicePrincipal'
  }
}

resource dsTest 'Microsoft.Resources/deploymentScripts@2023-08-01' = {
  name: '${prefix}DS'
  location: location
  identity: {
    type: 'userAssigned'
    userAssignedIdentities: {
      '${userAssignedIdentity.id}': {}
    }
  }
  kind: 'AzureCLI'
  properties: {
    forceUpdateTag: utcValue
    azCliVersion: '2.59.0'
    storageAccountSettings: {
      storageAccountName: storageAccountName
      storageAccountKey: null
    }
    containerSettings: {
      subnetIds: [
        {
          id: subnet.id
        }
      ]
    }
    scriptContent: 'echo "Hello world!"'
    retentionInterval: 'P1D'
    cleanupPreference: 'OnExpiration'
  }
  dependsOn: [
    roleAssignment, storageAccount
  ]
}

Additional context

The container instance is created, but it just sits in the 'Waiting' state.

[speters82]

The error was:

{"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/REDACTED/resourceGroups/rg_scripttest/providers/Microsoft.Resources/deployments/deployscripttest","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"ResourceDeploymentFailure","target":"/subscriptions/REDACTED/resourceGroups/rg_scripttest/providers/Microsoft.Resources/deploymentScripts/test2DS","message":"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.","details":[{"code":"DeploymentScriptACIProvisioningTimeout","message":"The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: REDACTED. Please try again later, if the issue persists contact technical support for further investigation."}]}]}}

[segraef]

Same here, as opposed to your config @speters82 I had to enable allowSharedKeyAccess which is by the way not obvious on the documentation because it's using the default value of allowSharedKeyAccess (Microsoft.Storage/storageAccounts@2023-01-01) which is true.

In addition as you can see in the below config I'm required to use a CMK-enabled SA which both are using private endpoints.

Configuration

targetScope = 'subscription'

@description('Optional. Location for all resources.')
param location string

@description('Optional. Resource group name for all resources.')
param resourceGroupName string

@description('Required. Tags for the resources.')
param tags object

@description('Required. Name of the user identity to create.')
param userIdentityName string = ''

@description('Optional. Name of the deployment script to create.')
param deployemntScriptName string = ''

@description('Required. Name of the key vault to create.')
param keyVaultName string = ''

@description('Required. Name of the key to create for encryption.')
param cmkKeyName string = ''

@description('Required. Dataclass of the key to create.')
param cmkKeyDataClass string = ''

@description('Required. Name of the storage account to create.')
param storageAccountName string = ''

@description('Required. Resource ID of the subnet to use for the private endpoints.')
param privateEndpointSubnetResourceId string = ''

@description('Optional. Resource ID of the subnet to use for the deployment script where the ACI gets placed into.')
param subnetResourceId string = ''

module resourceGroup 'ts/gml:resources.resource-group:0.1.2592' = {
  name: '${uniqueString(deployment().name, location)}-rg'
  params: {
    name: resourceGroupName
    location: location
    tags: tags
  }
}

module userIdentity 'ts/gml:managed-identity.user-assigned-identity:0.1.2530' = if (!empty(userIdentityName)) {
  scope: az.resourceGroup(resourceGroupName)
  name: '${uniqueString(deployment().name, location)}-uai'
  params: {
    name: userIdentityName
    location: location
    tags: tags
  }
}

module keyVault 'ts/gml:key-vault.vault:0.2.2545' = if (!empty(keyVaultName)) {
  scope: az.resourceGroup(resourceGroupName)
  name: '${uniqueString(deployment().name, location)}-kv'
  params: {
    name: keyVaultName
    location: location
    tags: tags
    privateEndpoints: [
      {
        name: 'pe-vault-01-${keyVaultName}'
        customNetworkInterfaceName: 'nic-pe-vault-01-${keyVaultName}'
        subnetResourceId: privateEndpointSubnetResourceId
      }
    ]
    enableVaultForTemplateDeployment: true
    enableVaultForDeployment: true
    networkAclsBypass: 'AzureServices'
    roleAssignments: [
      {
        principalId: userIdentity.outputs.principalId
        roleDefinitionIdOrName: 'Key Vault Crypto User'
      }
    ]
  }
}

module key 'ts/gml:key-vault.vault.key:0.1.2531' = if (!empty(cmkKeyName)) {
  scope: az.resourceGroup(resourceGroupName)
  name: '${uniqueString(deployment().name, location)}-key'
  params: {
    name: cmkKeyName
    tags: tags
    dataClass: cmkKeyDataClass
    keyVaultName: keyVault.outputs.name
    kty: 'RSA'
    attributesExp: 1725109032
  }
}

module storageAccount 'ts/gml:storage.storage-account:0.2.2557' = if (!empty(storageAccountName)) {
  scope: az.resourceGroup(resourceGroupName)
  name: '${uniqueString(deployment().name, location)}-sa'
  params: {
    location: location
    name: storageAccountName
    tags: tags
    networkAclsBypass: 'AzureServices'
    managedIdentities: {
      userAssignedResourceIds: [
        userIdentity.outputs.resourceId
      ]
    }
    customerManagedKey: {
      keyName: key.outputs.name
      keyVaultResourceId: keyVault.outputs.resourceId
      userAssignedIdentityResourceId: userIdentity.outputs.resourceId
    }
    privateEndpoints: [
      {
        name: 'pe-file-01-${storageAccountName}'
        customNetworkInterfaceName: 'nic-pe-file-01-${storageAccountName}'
        service: 'File'
        subnetResourceId: privateEndpointSubnetResourceId
      }
    ]
    roleAssignments: [
      {
        principalId: userIdentity.outputs.principalId
        roleDefinitionIdOrName: storageFileDataPrivilegedContributor.id
      }
    ]
  }
}

resource storageFileDataPrivilegedContributor 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: '69566ab7-960f-475b-8e7c-b3118f30c6bd'
  scope: tenant()
}

module deploymentScript 'br/public:avm/res/resources/deployment-script:0.2.0' = {
  scope: az.resourceGroup(resourceGroupName)
  name: '${uniqueString(deployment().name, location)}-ds'
  params: {
    name: deployemntScriptName
    location: location
    kind: 'AzurePowerShell'
    retentionInterval: 'P1D'
    subnetResourceIds: [
      subnetResourceId
    ]
    managedIdentities: {
      userAssignedResourcesIds: [
        userIdentity.outputs.resourceId
      ]
    }
    // runOnce: true
    scriptContent: 'echo \'Script test!\''
    storageAccountResourceId: storageAccount.outputs.resourceId
    azPowerShellVersion: '11.5'
  }
}

Deployment

Error

The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: f3906855-a9c2-4705-87ce-dc092149354f. Please try again later, if the issue persists contact technical support for further investigation. (Code: DeploymentScriptACIProvisioningTimeout)

[speters82]

My problem is that my company is rolling out a new security policy that requires SAS keys to be disabled on storage resources. I can somewhat work around it for now by using another script to enable the SAS key, run the deployment script that creates the keyvault certs, and then another one to disable the SAS key.

module enableSasKey './modules/deploymentStorageSasKey.bicep' = {
  name: 'enableSasKey'
  params: {
    enable: true
    identityId: adminManagedIdentity.id
    storageAccountName: deploymentStorageModule.outputs.storageAccountName
  }
}

// Our ingress certificate
module appGatewayIngressCert './modules/keyvaultCertificate.bicep' = {
  name: 'appGatewayIngressCert'
  params:{
    certName: stampResourceNames.appGatewayIngressCertName
    domainName: stampFqdn
    identityId: adminManagedIdentity.id
    issuer: 'OneCertV2-PublicCA'
    vaultName: keyvaultModule.outputs.keyVaultName
    subnetId: virtualNetwork.outputs.containerInstanceSubnetId
    storageAccountName: deploymentStorageModule.outputs.storageAccountName
  }
  dependsOn: [ deploymentStoragePrivateLink, keyvaultPrivateLink, enableSasKey ]
}

// SNIP: Other certs go here

module disableSasKey './modules/deploymentStorageSasKey.bicep' = {
  name: 'disableSasKey'
  params: {
    enable: false
    identityId: adminManagedIdentity.id
    storageAccountName: stampResourceNames.deploymentStorageName
  }
  dependsOn: [ orleansCert, genevaCert, appGatewayIngressCert ]
}

deploymentStorageSasKey:

param location string = resourceGroup().location

@description('ARM Id of the identity to execute the script')
param identityId string

@description('The name of the storage account to use to hold scripts')
param storageAccountName string

@description('Whether SAS Keys should be enabled or disabled in this account')
param enable bool

// This is because we want to force this to run every time we deploy this
param utcValue string = utcNow()

resource sasKeyScript 'Microsoft.Resources/deploymentScripts@2023-08-01' = {
  name: 'updateStorageSasKey'
  location: location
  identity: {
    type: 'userAssigned'
    userAssignedIdentities: {
      '${identityId}': {}
    }
  }
  kind: 'AzureCLI'
  properties: {
    forceUpdateTag: utcValue
    azCliVersion: '2.59.0'
    arguments: '${storageAccountName} ${resourceGroup().name} ${enable}'
    scriptContent: '''
      az storage account update --name $1 --resource-group $2 --allow-shared-key-access $3
    '''
    retentionInterval: 'PT1H'
    cleanupPreference: 'Always'
  }
}

That said, all of this feels completely unnecessary, and I should be able to just use an MSI on my script container to run this.

[anthony-c-martin]

@speters82 could you raise a support case for this?

[speters82]

I did: Support ticket 2406120010004253. I spent some time with the support engineer, and they weren't able to resolve the issue. They gave me some other resources, and I ended up having to find a way to work around this platform gap.

Thanks, Steve

Sent from Outlookhttp://aka.ms/weboutlook

---

From: Anthony Martin @.> Sent: Wednesday, June 19, 2024 1:24 PM To: Azure/bicep-types-az @.> Cc: Mention @.>; Author @.>; Comment @.***> Subject: Re: [Azure/bicep-types-az] Unable to run deployment script attached to storage account with AllowSharedKeyAccess false (Issue #2199)

@speters82https://github.com/speters82 could you raise a support case for this?

— Reply to this email directly, view it on GitHubhttps://github.com/Azure/bicep-types-az/issues/2199#issuecomment-2179415873 or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3CIHGT3M2TBAI7ZAQ54ZL3ZIHSI5BFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVI2TCMBZGY2DGNJXG2SG4YLNMWUWQYLTL5WGCYTFNSWHG5LCNJSWG5C7OR4XAZNMJFZXG5LFINXW23LFNZ2KM5DPOBUWG44TQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKJSHE4TMNZVG4YTPAVEOR4XAZNFNFZXG5LFUV3GC3DVMWVDEMZWGMYDONBXG43YFJDUPFYGLJLMMFRGK3FFOZQWY5LFVI2TCMBZGY2DGNJXG2TXI4TJM5TWK4VGMNZGKYLUMU. You are receiving this email because you were mentioned.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[alex-frankel]

I believe this is not a bug, but a feature that we are not yet able to support. @jorgecotillo - this is the same issue we are discussing internally right?

Assuming so, there are features that need to be implemented by both the Azure Storage team and Azure Container Instances team in order to be able to support this. My understanding is that work is underway, but probably too early for an ETA.