Resource Group reference not found with AKS role Assignment

[petegrimsdale]

Bicep version 0.3.1

Describe the bug When deploying and AKS cluster resource within a bicep template and using AAD pod identity there is a need to grant the Managed Machine Operator role to the kublet identity on the scope of the AKS resource group. The role assignment is done within bicep module. When this bicep file is built into an ARM template it generates and inner nested template with a scope defined as the aks node resource group as expected. When validating the bicep or ARM files however, the validation or deployment fails with an error stating the aks node resource group does not exist. If the resource group is created in advance then the aks deployment raises an error that the resource group already exists. Error on validate: {"error":{"code":"ResourceGroupNotFound","message":"Resource group 'rg-aksdemo2-aks' could not be found."}}

In ARM this error does not occur when the nested template is defined with outer scope

To Reproduce create a bicep template to deploy AKS resource and module to assign role to the aks node resource group scope - examples below

Additional context Example Bicep file:

param optionalSubnets array = []

@description('Specifies the CIDR notation IP range from which to assign pod IPs when kubenet is used.')
param podCidr string = ''

@description('A CIDR notation IP range from which to assign service cluster IPs. It must not overlap with any Subnet IP ranges.')
param serviceCidr string = ''

@description('Specifies the IP address assigned to the Kubernetes DNS service. It must be within the Kubernetes service address range specified in serviceCidr.')
param dnsServiceIP string = ''

@description('Specifies the CIDR notation IP range assigned to the Docker bridge network. It must not overlap with any Subnet IP ranges or the Kubernetes service address range.')
param dockerBridgeCidr string = ''

var resgpguid = substring(replace(guid(resourceGroup().id), '-', ''), 0, 4)
var uniqueResourceName_var = 'aksagic${resgpguid}'
var location = resourceGroup().location
var aksResourceGroup = '${resourceGroup().name}-aks'
var vnetName_var = '${uniqueResourceName_var}-vnet'
var dnsprefix = toLower(clusterName_var)
var defaultSubnetPrefix = '10.0.1.0/24'
var aksSubnetPrefix = '10.0.2.0/24'
var clusterName_var = '${uniqueResourceName_var}-aks'
var aksSubnetName = 'aks'
var aksSubnetId = resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName_var, aksSubnetName)
var standardSubnets = [
  'default'
  'aks'
]
var deploySubnets = concat(standardSubnets, optionalSubnets)
var subnets = [
  {
    name: 'default'
    addressPrefix: defaultSubnetPrefix
  }
  {
    name: 'aks'
    addressPrefix: aksSubnetPrefix
  }
]

resource uniqueResourceName 'Microsoft.OperationalInsights/workspaces@2020-03-01-preview' = {
  name: uniqueResourceName_var
  location: resourceGroup().location
  properties: {
    sku: {
      name: 'PerGB2018'
    }
  }
}

resource vnetName 'Microsoft.Network/virtualNetworks@2019-11-01' = {
  name: vnetName_var
  location: location
  tags: {
    displayName: vnetName_var
  }
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [for subnet in subnets: {
      name: subnet.name
      properties:{
        addressPrefix: subnet.addressPrefix
        privateLinkServiceNetworkPolicies: subnet.PrivateLinkServiceNetworkPolicies
      }
    }]
  }
}

resource clusterName 'Microsoft.ContainerService/managedClusters@2020-12-01' = {
  name: clusterName_var
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    dnsPrefix: dnsprefix
    nodeResourceGroup: aksResourceGroup
    agentPoolProfiles: [
      {
        name: 'system'
        count: 3
        enableAutoScaling: true
        maxCount: 10
        minCount: 3
        vmSize: 'Standard_DS3_v2'
        osType: 'Linux'
        mode: 'System'
        type: 'VirtualMachineScaleSets'
        availabilityZones: [
          '1'
          '2'
          '3'
        ]
        vnetSubnetID: aksSubnetId
      }
    ]
    addonProfiles: {
      omsagent: {
        enabled: true
        config: {
          logAnalyticsWorkspaceResourceID: uniqueResourceName.id
        }
      }
    }
    enableRBAC: true
    networkProfile: {
      networkPlugin: 'azure'
      networkPolicy: 'azure'
      podCidr: podCidr
      serviceCidr: serviceCidr
      dnsServiceIP: dnsServiceIP
      dockerBridgeCidr: dockerBridgeCidr
      loadBalancerSku: 'standard'
    }
    apiServerAccessProfile: {
      enablePrivateCluster: false
    }
  }
  dependsOn: []
}

module ClusterRoleAssignmentDeployment './nested_ClusterRoleAssignmentDeployment.bicep' = {
  name: 'ClusterRoleAssignmentDeployment'
  scope: resourceGroup(aksResourceGroup)
  params: {
    clustername: clusterName_var
    aksResourceGroup: aksResourceGroup
  }
  dependsOn: [
    clusterName
  ]
}

Module:

param clustername string
param aksResourceGroup string

resource id_clustername_ManagedOperatorResourceGroup 'Microsoft.Authorization/roleAssignments@2017-09-01' = {
  name: guid('${resourceGroup().id}${clustername}ManagedOperatorResourceGroup')
  properties: {
    roleDefinitionId: '/subscriptions/${subscription().subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/f1a07417-d97a-45cb-824c-7a7467783830'
    principalId: reference(resourceId('Microsoft.ContainerService/managedClusters', clustername), '2020-03-01').identityProfile.kubeletidentity.objectId
    scope: '/subscriptions/${subscription().subscriptionId}/resourceGroups/${aksResourceGroup}'
  }
}

resource id_clusterName_VMCResourceGroup 'Microsoft.Authorization/roleAssignments@2017-09-01' = {
  name: guid('${resourceGroup().id}${clustername}VMCResourceGroup')
  properties: {
    roleDefinitionId: '/subscriptions/${subscription().subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/9980e02c-c2be-4d73-94e8-173b1dc7cf3c'
    principalId: reference(resourceId('Microsoft.ContainerService/managedClusters', clustername), '2020-03-01').identityProfile.kubeletidentity.objectId
    scope: '/subscriptions/${subscription().subscriptionId}/resourceGroups/${aksResourceGroup}'
  }
}

[petegrimsdale]

Subsequently: This module definition fails to find the resource group for the aks nodes:

 param clustername string

var virtualMachineContributorRole = '/subscriptions/${subscription().subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/9980e02c-c2be-4d73-94e8-173b1dc7cf3c'

resource id_clusterName_VMCResourceGroup 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid('appKubeletIdentityVirtualMachineContributor', resourceGroup().id)
  scope: resourceGroup()
  properties: {
    roleDefinitionId: virtualMachineContributorRole
    principalId: reference(resourceId('Microsoft.ContainerService/managedClusters', clustername), '2020-03-01', 'Full').identityProfile.kubeletidentity.objectId
    principalType:'ServicePrincipal'
  }
}

However if the kubeletidentity.objectId is output from an AKS module and passed to the role assignment this works as:

param aksKubeletIdentityObjectId string

var virtualMachineContributorRole = '/subscriptions/${subscription().subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/9980e02c-c2be-4d73-94e8-173b1dc7cf3c'

resource appKubeletIdentityVirtualMachineContributor 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid('appKubeletIdentityVirtualMachineContributor', resourceGroup().id)
  scope: resourceGroup()
  properties: {
    roleDefinitionId: virtualMachineContributorRole
    principalId: aksKubeletIdentityObjectId
    principalType: 'ServicePrincipal'
  }
}

[alex-frankel]

I'm a bit lost on what is the latest code you are working with. Can you paste the latest files you are using? From your last statement, it sounds like you may have gotten things working.

[petegrimsdale]

@alex-frankel - I have this working now passing the principalId to the module as a parameter solves the issue. Same applies when using an "inner" nested ARM template.