microsoftgraph / msgraph-bicep-types

Repo contains Microsoft Graph resource types to integrate with bicep templates.
MIT License
45 stars 7 forks source link

Not a valid reference update error UserAssignedIdentity and Microsoft.Graph/appRoleAssignedTo@v1.0 #193

Open Anttarax opened 1 week ago

Anttarax commented 1 week ago

I want to create UserAssignedIdentity, After that I want to Assign an Appreg role to the newly created (service principal of) UserAssignedIdentity. The UserAssignedIdentity created successfully, but "Microsoft.Graph/appRoleAssignedTo@v1.0" thinks that userAssignedIdentity.properties.principalId is invalid, but not.

If I run second time, The deployment success, but first time, when the UserAssignedIdentity creating, I get this error message.

Error message: Not a valid reference update. Graph client request id: 370e13eb-3387-4e5a-b2b6-5f77c357b5dc. Graph request timestamp: 2024-11-18T00:13:49Z. (Code: BadRequest)

Calling userAssignedIdentities.bicep:

module userAssignedIdentity 'userAssignedIdentities.bicep' = if (userAssignIdentityrequired == true) {
  name: guid(appname, appRoleId)
  scope: resourceGroup(SubscriptionId, ResourceGroupName)
  params: {
    tags: tags
    appname: appname
    appRoleId: appRoleId
    AppServicePrincipal: AppServicePrincipal
  }
}

userAssignedIdentities.bicep:

param appname string
param tags object
extension microsoftGraph
param appRoleId string
param AppServicePrincipal string

resource userAssignedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' =  {
  name: 'id-${appname}'
  location: 'westeurope'
  tags: tags
}

@description('Grant role to the UserAssignedIdentity')
resource appRoleAssignedToUserAssignedIdentity 'Microsoft.Graph/appRoleAssignedTo@v1.0' = {
  appRoleId: appRoleId
  principalId: userAssignedIdentity.properties.principalId
  resourceId: AppServicePrincipal
}
dkershaw10 commented 1 week ago

@Anttarax - during deployment, what are you setting the resourceId and appRoleId to? I don't see anywhere that defines what AppServicePrincipal or what the AppRoleId values are. Are you providing these on the cmd line?

I'll have to check, but the "Not a valid reference" message can apply to any of the properties of the Microsoft.Graph/appRoleAssignedTo resource properties.

I'm not sure I follow a lot of the information, especially: If I run second time, The deployment success, but first time, when the UserAssignedIdentity creating, I get this error message. DONT USE THE SAME SCOPE FOR USERASSIGNEDIDENTITY, BECAUSE in that case there are no error message. But We dont want to create all UserAssignedIdentity in the same ResourceGroup, but the same Appreg Bicep create all of them, each to another ResourceGroup

Can you shed some more light on this please?

Anttarax commented 1 week ago

@dkershaw10 No, I dont providing these on the cmd line. But you should because its a simple way to test it.

I completely automated the creation of Appregs. I use all the Bicep Graph resources, a bicepparam contains all Appregs in an array and everything is generated.

I can share the complete code, but it's very long and it would take you several hours/days to analyze it. Just use a valid AppServicePrincipal and ApproleId, believe me the values ​​are good, as I generate more than 100 Appregs with this code every day. Now the code has only been extended by creating the UserAssignedIdentity (and role assign to the UserAssignedIdentity). System Assigned Managed Identity and Appreg role to Appreg work correctly. Please unit test it.

Képernyőfotó 2024-11-18 - 20 04 36
Anttarax commented 1 week ago

@dkershaw10 Here you are. A very simple code. Just replace the values of appRoleId and AppServicePrincipal to your own Appreg. You will definitely get an error. The scope and ResourceGroup that I mentioned earlier are not interesting, you will get an error anyway with or without any module scope (at first run, when userAssignedIdentity is not exist and creating.

extension microsoftGraph
param appname string = 'example-app' //Any string
param appRoleId string  = '91804f26-164c-53ec-b602-5f5e895fe73c' //ID of any Role in Appreg (same Appreg as in AppServicePrincipal)
param AppServicePrincipal string = '3135ce86-5184-465a-b1e9-1379e8b7df3d' //Enterprise Application Object ID of any Appreg (same Appreg as in appRoleId)

resource userAssignedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-07-31-preview' =  {
  name: 'id-${appname}'
  location: 'westeurope'
}

@description('Grant role to the UserAssignedIdentity')
resource appRoleAssignedToUserAssignedIdentity 'Microsoft.Graph/appRoleAssignedTo@v1.0' = {
  appRoleId: appRoleId
  principalId: userAssignedIdentity.properties.principalId
  resourceId: AppServicePrincipal
}

Output: Not a valid reference update. Graph client request id: f26c6d30-fecd-46c0-9b50-c66ce2571bdd. Graph request timestamp: 2024-11-19T04:07:25Z.\"}}"}]}}

dkershaw10 commented 6 days ago

Interesting. I have a similar script that I tried (below). It completes successfully when I deploy. I will also try referencing the created objects in a separate script, similar to above. I will likely need one of our engineers to look at our trace telemetry to see exactly what the issue might be.

extension graphv1_0

// TEMPLATE OVERVIEW: Creates a MI as a client service, registers a resource app which
// defines an app role, and then creates a resource app SP.
// Finally assign the app role (on the resource app SP) to the MI

param location string = resourceGroup().location

param deployDate string = '2024-11-19'
param clientAppName string = 'miService-${deployDate}'
param resourceName string = 'resourceServiceName-${deployDate}'

@description('Id of the application role to add to the resource app')
var appRoleName = 'ResourceAppData.Read.All'
var appRoleId = guid(resourceName, appRoleName)

// create a managed identity for the client service
resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'mi-${uniqueString(clientAppName,location)}'
  location: location
}

// create resource app
resource resourceApp 'Microsoft.Graph/applications@v1.0' = {
  uniqueName: resourceName
  displayName: resourceName
  appRoles: [
    {
      id: appRoleId
      allowedMemberTypes: [ 'User', 'Application' ]
      description: 'Read access to resource app data'
      displayName: appRoleName
      value: appRoleName
      isEnabled: true
    }
  ]
}

// Create resource SP
resource resourceSp 'Microsoft.Graph/servicePrincipals@v1.0' = {
  appId: resourceApp.appId
}

// Assign app role (from resource app) to MI
resource appRoleAssignment 'Microsoft.Graph/appRoleAssignedTo@v1.0' = {
  principalId: managedIdentity.properties.principalId
  resourceId: resourceSp.id
  appRoleId: appRoleId
}

output clientSpId string = managedIdentity.properties.principalId
output resourceSpId string = resourceSp.id
output appRoleId string = appRoleId
dkershaw10 commented 6 days ago

I managed to reproduce your error, but only once, having tried > 10 times with the following steps.

Repro steps

  1. Create a Bicep template that creates a user assigned managed identity, that also assigns an app role to the created managed identity - like your template, the resource app and role are pre-created and referenced by their IDs (although you could read them using the existing keyword too).
  2. Deploy the template.
  3. Wait a bit and redeploy the template.

Expected Both deployments are successful.

Actual (when it failed) First deployment fails with the "Not a valid reference update." error Second deployment succeeds.

Likely cause When creating a managed identity, this is created via Azure/ARM APIs (which under the hood create a service principal in Entra ID). When creating the app role assignment, this request is coming via the Graph Bicep extension calling Microsoft Graph and then on to Entra ID. It is possible that this second request hits a different read replica from the first request, and that the created service principal is not (yet) present in the read replica. I'll need to ask engineering to confirm if this might be the case.

@Anttarax - if you redeploy the same template file a little later, is the redeployment successful?

cc: @eketo-msft

Anttarax commented 6 days ago

@dkershaw10 Yes. As I mentioned at the very beginning, running it a second time, it succeeds when the userAssignedIdentity already exists. First step of Repro steps: Delete userAssignedIdentity if already exist. After that run the template.

slavizh commented 6 days ago

@dkershaw10 most likely it is the read replica thing. There is the same issue when creating system/user assigned identities and assigning RBAC roles (role assignments) to them. To overcome this role Assignments API https://learn.microsoft.com/en-us/azure/templates/microsoft.authorization/roleassignments?pivots=deployment-language-bicep has the parameter principalType which when provided value ServicePrincipal it does not check if the identity exists, it just assigns the permissions. Most likely the team behind role assignments knows the inner works details. Of course not suggesting that the same thing for role assignments will work or you have to implement it just mentioning that similar issue exists with other RPs.

dkershaw10 commented 6 days ago

@Anttarax - All my "re-runs" were with a new user-assigned MI creation.

@slavizh - yeah maybe the Azure role assignment has some special treatment here (could be because it doesn't want to call out to Entra to check, for perf reasons - who knows :)). Anyways, little we can do about the inner workings of the Entra system for app role assignment. One thing we have talked about internally (and I think we have a backlog item for it) is to have some retry logic in the Graph Bicep extension. I don't know that we want to do that for every 400 error, but maybe for some specific error conditions, like 400s from this appRoleAssignedTo API, as this seems like a common Azure -> Entra crossover case. Another one might be when creating a federated identity credential using a newly created managed identity as the subject.

@eketo-msft what do you think?