Pulumi can't retrieve the client token from a managed identity on an Azure-hosted VM.

[stooj]

What happened?

On a hosted VM with a Managed Identity, pulumi can't retrieve the client token.

Running the following inside pulumi:

var token = await GetClientToken.InvokeAsync();

crashes with the following:

 Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="invocation of azure-native:authorization:getClientToken returned an error: empty token from *autorest.BearerAuthorizer")
       at async Task<InvokeResponse> Pulumi.GrpcMonitor.InvokeAsync(ResourceInvokeRequest request)
       at async Task<SerializationResult> Pulumi.Deployment.InvokeRawAsync(string token, SerializationResult argsSerializationResult, InvokeOptions options) x 2
       at async Task<T> Pulumi.Deployment.InvokeAsync<T>(string token, InvokeArgs args, InvokeOptions options, bool convertResult)
       at async Task Program.<Main>$(string[] args)+(?) => { } in C:/Users/jason/Documents/pulumi-test/Program.cs:line 13
       at async Task<int> Pulumi.Deployment.RunAsync(Func<Task> func)+(?) => { }
       at async Task<IDictionary<string, object>> Pulumi.Stack.RunInitAsync(Func<Task<IDictionary<string, object>>> init)
       at async Task<OutputData<T>> Pulumi.Output<T>+<>c__DisplayClass12_0.<Create>g__GetData|0(?)+GetData(?)
       at async Task<T> Pulumi.Output<T>.GetValueAsync(T whenUnknown)
       at async Task Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output<IDictionary<string, object>> outputs)

Example

Here's a sample program to trigger the issue:

using System;
using Pulumi.AzureNative.Authorization;
using Pulumi.AzureNative.Resources;
using Deployment = Pulumi.Deployment;

await Deployment.RunAsync(async () =>
{
    var token = await GetClientToken.InvokeAsync();

    Console.WriteLine($"Token: {token.Token}");
});

Here's a separate pulumi project to bootstrap a test environment in Azure:

Sample csproj file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Pulumi" Version="3.*" />
    <PackageReference Include="Pulumi.AzureNative" Version="2.*" />
    <PackageReference Include="Pulumi.Random" Version="4.*" />
    <PackageReference Include="Pulumi.Tls" Version="4.*" />
  </ItemGroup>

</Project>

Pulumi code:

using Pulumi;
using AzureNative = Pulumi.AzureNative;
using Pulumi.AzureNative.Authorization;
using Pulumi.AzureNative.Authorization.Inputs;
using Pulumi.AzureNative.Compute;
using Pulumi.AzureNative.Compute.Inputs;
using Random = Pulumi.Random;
using System.Collections.Generic;
using System;
using System.Text;

return await Deployment.RunAsync(() => {
        // Import the program's configuration settings.
        var config = new Pulumi.Config();
        var vmName = config.Get("vmName") ?? "my-server";
        var vmSize = config.Get("vmSize") ?? "Standard_B1s";
        var osImage = config.Get("osImage") ?? "Debian:debian-11:11:latest";
        var adminUsername = config.Get("adminUsername") ?? "pulumiuser";
        var servicePort = config.Get("servicePort") ?? "80";
        var sshPubKey = config.Require("sshPubKey");

        var azureConfig = new Pulumi.Config("azure-native");
        var location = azureConfig.Get("location") ?? "eastus";
        var subscriptionId = azureConfig.Require("subscriptionId");

        string[] osImageArgs = osImage.Split(":");
        var osImagePublisher = osImageArgs[0];
        var osImageOffer = osImageArgs[1];
        var osImageSku = osImageArgs[2];
        var osImageVersion = osImageArgs[3];

        var roleDefinitionId = "/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c"; // Contributor role definition ID

        // Create a resource group
        var resourceGroup = new AzureNative.Resources.ResourceGroup("stoo-rg-msi-test", new()
                {
                Location = location,
                });
        var userAssignedIdentity = new AzureNative.ManagedIdentity.UserAssignedIdentity("userAssignedIdentity", new()
                {
                Location = resourceGroup.Location,
                ResourceGroupName = resourceGroup.Name,
                ResourceName = "stoo-uaid-msi-test",
                });
        var roleAssignment = new RoleAssignment("roleAssignment", new RoleAssignmentArgs
                {
                    RoleAssignmentName = Guid.NewGuid().ToString(),
                    Scope = $"/subscriptions/{subscriptionId}",
                    RoleDefinitionId = roleDefinitionId,
                PrincipalId = userAssignedIdentity.PrincipalId,
                PrincipalType = PrincipalType.ServicePrincipal
                });
        // Create a virtual network
        var virtualNetwork = new AzureNative.Network.VirtualNetwork("network", new()
                {
                ResourceGroupName = resourceGroup.Name,
                AddressSpace = new AzureNative.Network.Inputs.AddressSpaceArgs {
                AddressPrefixes = new[]
                {
                "10.0.0.0/16",
                },
                },
                Subnets = new[] {
                new AzureNative.Network.Inputs.SubnetArgs {
                Name = $"{vmName}-subnet",
                AddressPrefix = "10.0.1.0/24",
                },
                },
                });

        // Use a random string to give the VM a unique DNS name
        var domainNameLabel = new Random.RandomString("domain-label", new()
                {
                Length = 8,
                Upper= false,
                Special = false,
                }).Result.Apply(result => $"{vmName}-{result}");

        // Create a public IP address for the VM
        var publicIp = new AzureNative.Network.PublicIPAddress("public-ip", new()
                {
                ResourceGroupName = resourceGroup.Name,
                PublicIPAllocationMethod = AzureNative.Network.IPAllocationMethod.Dynamic,
                DnsSettings = new AzureNative.Network.Inputs.PublicIPAddressDnsSettingsArgs {
                DomainNameLabel = domainNameLabel,
                },
                });

        // Create a security group allowing inbound access over ports 80 (for HTTP) and 22 (for SSH)
        var securityGroup = new AzureNative.Network.NetworkSecurityGroup("security-group", new()
                {
                ResourceGroupName = resourceGroup.Name,
                SecurityRules = new[]
                {
                new AzureNative.Network.Inputs.SecurityRuleArgs {
                Name = $"{vmName}-securityrule",
                Priority = 1000,
                Direction = AzureNative.Network.SecurityRuleDirection.Inbound,
                Access = "Allow",
                Protocol = "Tcp",
                SourcePortRange = "*",
                SourceAddressPrefix = "*",
                DestinationAddressPrefix = "*",
                DestinationPortRanges = new[]
                {
                servicePort,
                "22",
                },
                },
                },
                });

        // Create a network interface with the virtual network, IP address, and security group
        var networkInterface = new AzureNative.Network.NetworkInterface("network-interface", new()
                {
                ResourceGroupName = resourceGroup.Name,
                NetworkSecurityGroup = new AzureNative.Network.Inputs.NetworkSecurityGroupArgs {
                Id = securityGroup.Id
                },
                IpConfigurations = new[]
                {
                new AzureNative.Network.Inputs.NetworkInterfaceIPConfigurationArgs {
                Name = $"{vmName}-ipconfiguration",
                PrivateIPAllocationMethod = AzureNative.Network.IPAllocationMethod.Dynamic,
                Subnet = new AzureNative.Network.Inputs.SubnetArgs {
                Id = virtualNetwork.Subnets.GetAt(0).Apply(subnet => subnet.Id!)
                },
                PublicIPAddress = new AzureNative.Network.Inputs.PublicIPAddressArgs {
                Id = publicIp.Id,
                },
                }
                }
                });

        // Define a script to be run when the VM starts up
        var initScript = $@"#!/bin/bash
            sudo apt-get update
            sudo apt-get install --assume-yes dotnet6
            if ! [[ -e {adminUsername}/.pulumi ]]; then
                sudo -u {adminUsername} bash -c 'curl -fsSL https://get.pulumi.com | sh'
            fi";

        // Create the virtual machine
        var vm = new AzureNative.Compute.VirtualMachine("vm", new()
                {
                ResourceGroupName = resourceGroup.Name,
                NetworkProfile = new AzureNative.Compute.Inputs.NetworkProfileArgs {
                NetworkInterfaces = new[] {
                new AzureNative.Compute.Inputs.NetworkInterfaceReferenceArgs {
                Id = networkInterface.Id,
                Primary = true,
                },
                },
                },
                Identity = new VirtualMachineIdentityArgs
                {
                    Type = AzureNative.Compute.ResourceIdentityType.UserAssigned,
                    // We don't mirror Azure here. We use an InputList
                    UserAssignedIdentities = new InputList<string> { userAssignedIdentity.Id },
                },
                HardwareProfile = new AzureNative.Compute.Inputs.HardwareProfileArgs {
                    VmSize = vmSize,
                },
                OsProfile = new AzureNative.Compute.Inputs.OSProfileArgs {
                    ComputerName = vmName,
                    AdminUsername = adminUsername,
                    CustomData = Convert.ToBase64String(Encoding.UTF8.GetBytes(initScript)),
                    LinuxConfiguration = new AzureNative.Compute.Inputs.LinuxConfigurationArgs {
                        DisablePasswordAuthentication = true,
                        Ssh = new AzureNative.Compute.Inputs.SshConfigurationArgs {
                            PublicKeys = new[]
                            {
                                new AzureNative.Compute.Inputs.SshPublicKeyArgs {
                                    KeyData = sshPubKey,
                                    Path = $"/home/{adminUsername}/.ssh/authorized_keys",
                                },
                            },
                        },
                    },
                },
                StorageProfile = new AzureNative.Compute.Inputs.StorageProfileArgs {
                    OsDisk = new AzureNative.Compute.Inputs.OSDiskArgs {
                        Name = $"{vmName}-osdisk",
                        CreateOption = AzureNative.Compute.DiskCreateOptionTypes.FromImage,
                    },
                    ImageReference = new AzureNative.Compute.Inputs.ImageReferenceArgs {
                        Publisher = osImagePublisher,
                        Offer = osImageOffer,
                        Sku = osImageSku,
                        Version = osImageVersion,
                    },
                },
                });

        // Once the machine is created, fetch its IP address and DNS hostname
        var vmAddress = vm.Id.Apply(_ => {
                return AzureNative.Network.GetPublicIPAddress.Invoke(new()
                        {
                        ResourceGroupName = resourceGroup.Name,
                        PublicIpAddressName = publicIp.Name,
                        });
                });

        // Export the VM's hostname, public IP address, HTTP URL, and SSH private key
        return new Dictionary<string, object?>
        {
            ["hostname"] = vmAddress.Apply(addr => addr.DnsSettings!.Fqdn),
                ["ip"] = vmAddress.Apply(addr => addr.IpAddress),
                ["url"] = vmAddress.Apply(addr => $"http://{addr.DnsSettings!.Fqdn}:{servicePort}"),
                ["principalId"] = userAssignedIdentity.PrincipalId,
        };
        });

To reproduce, ssh into the VM and start a pulumi program using useMsi or the ARM_USE_MSI env variable.

mkdir ~/.pulumi-creds
pulumi login file://~/.pulumi-creds
mkdir test && cd test
export ARM_USE_MSI=true
pulumi new azure-csharp -y
pulumi config set azure-native:subscriptionId <get from `az account show` on dev machine>

Output of pulumi about

NAME                VERSION
Pulumi              3.63.1
Pulumi.AzureNative  2.41.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[danielrbradley]

Here's my initial analysis ...

This error is either coming from:

• getAuthConfig
• getOAuthToken

It must be the second, because getting auth config: doesn't appear in the error message.

The definition for getOAuthToken which contains the error string empty token from %T. Here's the logic to fetch the token:

https://github.com/pulumi/pulumi-azure-native/blob/44445c318106d1937c6e87d3b8a34f9ee292f10a/provider/pkg/provider/auth.go#L302-L315

This is trying to cast the authorizer (autorest.Authorizer) into either a autorest.BearerAuthorizer or hamiltonAuth.Authorizer. When using MSI, we should be using the managedServiceIdentityAuth method in go-azure-helpers. It appears this is returning a BearerAuthorizer so should be correct.

[thomas11]

@stooj, does the VM have a system- or user-assigned identity? Can you double check your clientId config - it should be unset for system identity and set for user identity.

[stooj]

I was using a User Assigned Identity.

I've got the whole recreation in the initial issue description for recreating from start to finish. I'm not setting clientId anywhere manually and I'm not manually logging into azure once I've created the VM.

var userAssignedIdentity = new AzureNative.ManagedIdentity.UserAssignedIdentity("userAssignedIdentity", new()
                {
                Location = resourceGroup.Location,
                ResourceGroupName = resourceGroup.Name,
                ResourceName = "stoo-uaid-msi-test",
                });

My instructions were maybe misleading there because I do run az account show on my local dev machine to retrieve the subscriptionId

[thomas11]

I think our docs might be wrong here, and you do need to configure clientId for user-managed identities. You should be able to find the id of your identity in the portal. Can you give that a try?

[stooj]

Sure thing.

Tried to set the clientId using an env var and in the pulumi config, but I still got the same error.

The original reporter is using a System Managed Identity as well; I used a User Managed one because it was easier to set up, but they both show the same error.

[stooj]

What I didn't set was the tenantId on the VM. After configuring that I was able to create a StorageAccount from inside the VM.

[stooj]

Just to complete the (closed) loop here, this was fixed somewhere.

Using latest pulumi latest (3.122.0 at writing) I'm able to retrieve the client token in my test azure vm, even if tenantId is not set.

[tomachristian]

This still does not work for us with the latest Pulumi version as of this writing.