AKS Engine deployment through MSI

[amankohli]

Hi,

I am trying to deploy the AKS engine through MSI with below deployment; AKS Engine deployment through MSI

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.13",
      "kubernetesConfig": {
        "useManagedIdentity": true,
        "userAssignedID": "testk8s",
        "networkPlugin": "kubenet",
        "privateCluster": {
          "enabled": true
        }
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "k8seastusdc",
      "vmSize": "Standard_D2s_v3",
      "OSDiskSizeGB": 100,
      "vnetSubnetId":"xx",
      "firstConsecutiveStaticIP": "10.xx.0.50"
    },
    "agentPoolProfiles": [
      {
        "name": "node",
        "count": 3,
        "vmSize": "Standard_D8s_v3",
        "OSDiskSizeGB": 100,
        "vnetSubnetId": "xx"
      }
    ],
    "linuxProfile": {
      "adminUsername": "ubuntu",
      "ssh": {
        "publicKeys": [
          {
            "keyData": ""
          }
        ]
      }
    }
  }

I have tried the deployment with the above tempate and get the below error: rror: resources.DeploymentsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidTemplateDeployment" Message="The template deployment failed with error: 'Authorization failed for template resource

Below is the document we were referring too: https://github.com/Azure/aks-engine/blob/master/examples/kubernetes-msi-userassigned/kube-vmss.json

[CecileRobertMichon]

Hi @amankohli, it looks like the deployment error you pasted above is incomplete, can you please paste the full error? Also, any reason you're using kubernetes 1.13? That's quite old and is not supported anymore by latest AKS Engine versions. What version of AKS Engine are you using to generate the template?

[amankohli]

Hi @CecileRobertMichon I tried the same installation of aks engine wiht 1.15 version:

"apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorRelease": "1.15", "kubernetesConfig": { "useManagedIdentity": true, "userAssignedID": "testk8s", "networkPlugin": "kubenet", "privateCluster": { "enabled": true } } }, "masterProfile": { "count": 1, "dnsPrefix": "k8seastusdc", "vmSize": "Standard_D2s_v3", "OSDiskSizeGB": 100, "vnetSubnetId":"xx", "firstConsecutiveStaticIP": "10.xx.0.50" }, "agentPoolProfiles": [ { "name": "node", "count": 3, "vmSize": "Standard_D8s_v3", "OSDiskSizeGB": 100, "vnetSubnetId": "xx" } ], "linuxProfile": { "adminUsername": "ubuntu", "ssh": { "publicKeys": [ { "keyData": "" } ] } } }

Below is the error: Error: resources.DeploymentsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidTemplateDeployment" Message="The template deployment failed with error: 'Authorization failed for template resource 'e' of type 'Microsoft.Authorization/roleAssignments'. The client '82' with object id '' does not have permission to perform action 'resourcid'

Note: The MSI has owner rights to the resource group where we are trying to deploy aks engine.

[CecileRobertMichon]

To be clear is "userAssignedID": "testk8s", the actual value you are using or did you redact the resource ID?

From the docs,

When `useManagedIdentity` is set to true, this string value indicates that user assigned identity will be the type of managed identity used for cluster nodes, and appropriate pods. If the string value of `"userAssignedID"` is a fully qualified resource ID (e.g., `"/subscriptions/7a8f2518-7462-11ea-bc55-0242ac130003/resourceGroups/my-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my-user-assigned-identity"`), then the cluster will re-use that pre-existing user assigned managed identity resource; if the string value of `"userAssignedID"` is a simple string (e.g., `"my-new-user-assigned-identity"`), then a new user assigned managed identity resource will be created in the cluster resource group, with a name that matches that string value.

Are you trying to reuse an existing identity resource or have aks-engine create one for you?

[amankohli]

@CecileRobertMichon Thank you the MSI deployment is working now after the service principal was made the owner of the resource group. Can we use MSI while invoking the aks engine command? Below are the steps we tried and didn't work: i) az login --identity ii) aks-engine deploy --subscription-id subid --location eastasia --api-model k8s.json -g rg --auth-method cli We get below error: CloudShell Error: parsing time "N/A" as "2006-01-02T15:04:05Z07:00": cannot parse "N/A" as "2006"

CLI Error: parsing time "N/A" as "2006-01-02 15:04:05.999999": cannot parse "N/A" as "2006"

[devigned]

@amankohli we don't currently have test using Managed Identity to provision, but I don't see why, given the proper RBAC for the identity, one could not use Managed Identities.

Can you explain a bit more about the CloudShell error and the CLI error?

• Are you seeing the error when you run az login --identity.
• Is the CLI error from a CloudShell instance? Is it from a provisioned VM with a System Assigned Identity?

Please add any more details you might think are pertinent.

[amankohli]

@devigned We are using user assigned identities which are attached to the VM. I don't see any error while running az login --identity. The managed identity is owner of the resource group.

[devigned]

Heads up with az login --identity, it will choose the first User Assigned Identity on the machine if a client ID is not specified.

Per the CLI error, is that error raised when running aks-engine? Just want to be clear.

[devigned]

My inclination is that the failure is coming from parsing the token returned by az. I think the tokens returned from IMDS have a slightly different format from AAD direct tokens.

https://github.com/Azure/aks-engine/blob/fce292453c194c3f2848eac210e017fbbca02b72/pkg/armhelpers/azureclient.go#L90

[amankohli]

@devigned can we add a feature request to authenticate using MSI? Below is the MS go documentation: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token#get-a-token-using-go

[devigned]

Hmm... I just created an Azure VM with a user assigned identity (role: subscription owner), which I used to create a user assigned identity enabled cluster.

davidjustice@testvm:~$ az login --identity -u e98f1110-1892-42c5-8cab-9ddef33f27f6

[
  {
    "environmentName": "AzureCloud",
    "homeTenantId": "guid-home-tenant-id",
    "id": "guid",
    "isDefault": true,
    "managedByTenants": [],
    "name": "extremis",
    "state": "Enabled",
    "tenantId": "guid-tenant-id",
    "user": {
      "assignedIdentityInfo": "MSIClient-e98f1110-1892-42c5-8cab-9ddef33f27f6",
      "name": "userAssignedIdentity",
      "type": "servicePrincipal"
    }
  }
]

davidjustice@testvm:~$ ./aks-engine deploy -m kubernetes.json -l westus2 --auth-method cli -f
INFO[0000] No subscription provided, using selected subscription from azure CLI: sub-guid
WARN[0002] --resource-group was not specified. Using the DNS prefix from the apimodel as the resource group name: footest123
INFO[0014] Starting ARM Deployment footest123-554824839 in resource group footest123. This will take some time...
INFO[0267] Finished ARM Deployment (footest123-554824839). Succeeded

Next Questions:

• What version of aks-engine are you using?
• Can you try specifying the client ID for the given user assigned identity you are attempting to use?

FYI: the test above was run via master@aks-engine

[amankohli]

The issue was with the older version of azure cli which was installed on the Vm rather than Aks enginer. The above command worked after I updated az cli version on the vm.