What's the correct way of providing a managed identity to an Azure VMSS created by a ManagedCluster?

[markphillips100]

Apologies if this has been asked before but I'm not seeing specific examples of what I'm trying to do. I'll explain.

Some background first. I'm new to azure-native having used the azure wrapper a year or so ago with time off till now. So in the past I would have created an Azure AKS cluster and installed aad-pod-identity to provide managed identity to pods. At the time aad-pod-identity was installed in what is now called "standard mode". This required creation of user managed identities that would be assigned to VMSS by a MIC controller. And in my particular use case I would create a user managed identity per application that I deploy to the cluster. This identity would be given whatever other AAD roles are required for it to access other Azure resources the app is dependent on. In short, the application can only access what is should.

Now the preference is to use "managed" mode with aad-pod-identity, and as I understand things this installation is handled automatically by enabling the feature when creating the ManagedCluster.

My question comes in here as to how application-specific user managed identities should be created and assigned to the cluster? I understand there's an azure cli command for setting against a vmss but as azure-native is supposed to be close to rest api as possible I figured there would be something I could use here.

I did see this issue which describes setting the "identity" property but I'm assuming that's specifying the property on the ManagedCluster resource as part of its creation. This made me think that there's only meant to be one user managed identity for the whole cluster? If so, that would mean all apps use the same identity which feels wrong.

In my case, I could have several different application deployments each with their own identity, so how to manage those identity assignments against the cluster/vmss without effecting any other deployment's identity?

[mikhailshilkov]

Do you have an example of what you are trying to achieve expressed in ARM Templates or Azure CLI?

[markphillips100]

I've only seen an example of this done using Azure Cli as described here on the aad-pod-identity docs, although I've not tried this myself yet.

This is the specific command:

az vmss identity assign -g <VM resource group name> -n <VMSS name> --identities <resource ID of managed identity>

I can make the VMSS name and VM resource group name outputs of my cluster deployment, assuming there's a way of obtaining the name of the cluster's resource group using @pulumi/azure-native?

Assuming the above is possible, is it possible to achieve the cli command's behaviour purely with @pulumi/azure-native?

[markphillips100]

Any ideas yet how I achieve the identity assignment?

[mikhailshilkov]

May be a naive question, but isn't this achieved with the userAssignedIdentities property of the managed cluster?

[mikhailshilkov]

It would be great if you could achieve what you need once (on a test environment) with Azure portal or CLI, then export the ARM template of your resources and see what exactly that command changed. If it's all in ARM API surface, this should be doable with Pulumi resources.

[markphillips100]

So is this the general direction Pulumi is now taking? I mean now I have to understand ARM to work out what I'm supposed to do using Pulumi azure native?

[markphillips100]

May be a naive question, but isn't this achieved with the userAssignedIdentities property of the managed cluster?

It may be the correct way but I don't have an example to follow. And as I described before this then implies that the responsibility of provisioning the required app-specific identities now lies with the cluster provisioning. This feels incorrect or at least causes friction when needing app identity isolation. For example, I wouldn't want to update my cluster project to add a new identity when it should really be done by my app's infrastructure project.

I guess I need to find more information on what the general direction is for clusters and app user identities. And as you pointed out, it might be easier to determine a portal/cli setup first and then worry about the mechanics of the api calls.

[markphillips100]

According to the aad-pod-identity documentation for managed-mode user identities for pods the identity must be assigned to the required VMSS userAssignedIdentities, not the the cluster's userAssignedIdentities. Here is the ARM template for doing that.

This is the template:

{
    "name": "[variables('vmssName')]",
    "apiVersion": "2018-06-01",
    "location": "[parameters(Location')]",
    "identity": {
        "type": "userAssigned",
        "userAssignedIdentities": {
            "[resourceID('Microsoft.ManagedIdentity/userAssignedIdentities/',variables('<USERASSIGNEDIDENTITYNAME>'))]": {}
        }
    }
}

Is this api possible with pulumi native?

[markphillips100]

I think the type is Microsoft.Compute/virtualMachineScaleSets .

[markphillips100]

So then I think I add the required identity to the userAssignedIdentities described in the Pulumi docs here.

How does Pulumi handle the scenario of 2 different pulumi projects adding their own identity resource to this same property dictionary? Does it handle removal and insertions without affecting the other?

[mikhailshilkov]

I mean now I have to understand ARM to work out what I'm supposed to do using Pulumi azure native?

You need to understand Azure and its resource model (i.e. ARM) to understand which resources and properties to use in Pulumi. I'm not sure how it would be possible to avoid this requirement.

How does Pulumi handle the scenario of 2 different pulumi projects adding their own identity resource to this same property dictionary?

In this scenario, VMSS is created behind the scenes by AKS control plane, so you can't manage it from Pulumi. I guess you could import it and then update its properties but I expect this to be somewhat painful. Does Microsoft really require you to update resources which are managed by their control plane? This sounds like a bad idea to me. Will they never update that resource on your behalf and reset your changes?

[markphillips100]

You need to understand Azure and its resource model (i.e. ARM) to understand which resources and properties to use in Pulumi. I'm not sure how it would be possible to avoid this requirement.

Fair point. I meant no disrespect. I guess I was just highlighting that with the non-native pulumi lib I would just look at pulumi docs to understand capability.

In this scenario, VMSS is created behind the scenes by AKS control plane, so you can't manage it from Pulumi. I guess you could import it and then update its properties but I expect this to be somewhat painful. Does Microsoft really require you to update resources which are managed by their control plane? This sounds like a bad idea to me. Will they never update that resource on your behalf and reset your changes?

This was my worry, and your question about whether this is the intent of Microsoft or not is yet to be determined I suppose. It's also perhaps going to change come early next year when they bump the aad-pod-identity to v2 (yet to be finished).

Certainly for now the only docs available suggest you do indeed update the VMSS outside of the creation of the cluster. I might put a question up on the AKS repo for some clarity.

Thanks for your input so far @mikhailshilkov