Vault Agent doesn't support Azure Container Instances (ACI) with Managed Identities (MSI)

Describe the bug We're trying to use vault-agent as a sidecar container with Azure Container Instances with a User-assigned Managed Identity (for authentication/certification management) however, the vault-agent hits the Azure IMDS endpoint (http://169.254.169.254/metadata/instance) which we understand is meant only for VMs and hence encounters a 404 during it's agent initialization.

To Reproduce Deploy an Azure Container Instance (multi container) using a YAML Manifest:

containerinstance-manifest.yaml:

apiVersion: 2019-12-01
location: westus2
name: rohit-shenoy-Vault-ACI-App2
identity: 
  type: UserAssigned
  userAssignedIdentities: {'/subscriptions/<subscriptionId..........-42f1-......-......>/resourcegroups/revanth-vault-poc/providers/Microsoft.ManagedIdentity/userAssignedIdentities/revanth-vault-poc':{
      'clientId': ".....-666f-....-....-........",
      'principalId': ".........-eee7-.....-.....-......"
    }
  }
properties:
  containers:
  - name: album-api
    properties:
      image: rohitshenoycontainerregistry.azurecr.io/album-api:latest
      resources:
        requests:
          cpu: 1
          memoryInGb: 1.5
      ports:
      - port: 3500
  - name: vault
    properties:
      image: rohitshenoycontainerregistry.azurecr.io/vault-agent:latest
      environmentVariables:
        - name: VAULT_AGENT_EXIT_AFTER_AUTH
          value: false
        - name: AUTH_MOUNT_PATH
          value: 'auth/azure'
        - name: VAULT_ROLE
          value: 'container-instance-poc-role'
      resources:
        requests:
          cpu: 1
          memoryInGb: 1.5
  osType: Linux
  ipAddress:
    type: Public
    ports:
    - protocol: tcp
      port: 3500
  imageRegistryCredentials:
    - server: rohitshenoycontainerregistry.azurecr.io
      username: rohitshenoycontainerregistry
      password: <password>
tags: {applicationType: vault-container-instance-poc}
type: Microsoft.ContainerInstance/containerGroups

vault-agent Dockerfile:

FROM vault:1.13.1@sha256:17e6bd0e4c861200f982f2620b209215d997d427a14649466da3702a1a7e7151 as vault
FROM alpine:3.16.3@sha256:3d426b0bfc361d6e8303f51459f17782b219dece42a1c7fe463b6014b189c86d

COPY --from=vault /bin/vault /bin/vault
COPY agent.hcl /vault-agent/agent.hcl
COPY entrypoint.sh ./entrypoint.sh

RUN mkdir -p /config
RUN mkdir -p /vault-agent
RUN mkdir -p /pid
RUN mkdir -p /vault/secrets

ENTRYPOINT [ "sh", "./entrypoint.sh" ]

Deploy:

az provider register --namespace Microsoft.App

RESOURCE_GROUP="album-containerapps"
LOCATION="westus2"      
ENVIRONMENT="env-album-containerapps"
API_NAME="album-api"
FRONTEND_NAME="album-ui"

# create RG
az group create \
  --name $RESOURCE_GROUP \
  --location "$LOCATION"

GITHUB_USERNAME=rohitshenoy
ACR_NAME=$GITHUB_USERNAME"containerregistry"

# create container registry
az acr create \
  --resource-group $RESOURCE_GROUP \
  --name $ACR_NAME \
  --sku Basic \
  --admin-enabled true

# build application container dockerfile
/Users/rohit.shenoy/Documents/GitHub/approle/vault-poc/code-to-cloud/src> az acr build --registry $ACR_NAME --image $API_NAME .

# build vault agent dockerfile
/Users/rohit.shenoy/Documents/GitHub/approle/vault-poc/azure-vault-sidecar/vault-ecs-sidecar> az acr build --registry $ACR_NAME --image vault-agent .

# deploy
az container create --resource-group $RESOURCE_GROUP --file containerinstance-manifest.yaml

Error: (vault agent logs)

==> Note: Vault Agent version does not match Vault server version. Vault Agent version: 1.13.1, Vault server version: 1.13.0
==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://0.0.0.0:8200
                     Cgo: disabled
               Log Level: 
                 Version: Vault v1.13.1, built 2023-03-23T12:51:35Z
             Version Sha: 4472e4a3fbcc984b7e3dc48f5a8283f3efe6f282

2023-04-07T20:28:49.086Z [INFO]  agent.sink.file: creating file sink
2023-04-07T20:28:49.090Z [INFO]  agent.sink.file: file sink configured: path=/tmp/sink_file.txt mode=-rw-r--r--
2023-04-07T20:28:49.101Z [INFO]  agent.template.server: starting template server
2023-04-07T20:28:49.103Z [INFO]  agent.template.server: no templates found
2023-04-07T20:28:49.104Z [INFO]  agent.auth.handler: starting auth handler
2023-04-07T20:28:49.104Z [INFO]  agent.auth.handler: authenticating
2023-04-07T20:28:49.107Z [INFO]  agent.sink.server: starting sink server
2023-04-07T20:28:49.115Z [ERROR] agent.auth.handler: error getting path or data from method:
  error=
  | error response in metadata from http://169.254.169.254/metadata/instance: 404 page not found
   backoff=1s
2023-04-07T20:28:50.116Z [INFO]  agent.auth.handler: authenticating
2023-04-07T20:28:50.118Z [ERROR] agent.auth.handler: error getting path or data from method:
  error=
  | error response in metadata from http://169.254.169.254/metadata/instance: 404 page not found
   backoff=1.87s
2023-04-07T20:28:51.997Z [INFO]  agent.auth.handler: authenticating
2023-04-07T20:28:51.999Z [ERROR] agent.auth.handler: error getting path or data from method:
  error=
  | error response in metadata from http://169.254.169.254/metadata/instance: 404 page not found
   backoff=3.23s

As you can see, the vault-agent attempts to hit the http://169.254.169.254/metadata/instance endpoint and delivers a 404.

We believe this is the problematic code: https://github.com/hashicorp/vault/blob/main/command/agent/auth/azure/azure.go#L112

Expected behavior vault agent should hit the metadata/identity/oauth2/token endpoint, and perform a successful login.

Environment:

Vault Server Version (retrieve with vault status):

azureuser@revanth-poc3-vault-vm:~$ vault status
Key                      Value
---                      -----
Version                  1.13.0

Vault Agent Version (retrieve with vault version):

vault:1.13.1@sha256:17e6bd0e4c861200f982f2620b209215d997d427a14649466da3702a1a7e7151

Server Operating System/Architecture: Azure Container Instances

Vault agent configuration file(s):

exit_after_auth = VAULT_AGENT_EXIT_AFTER_AUTH
pid_file        = "/pid/pidfile"

vault {
  address = "http://20.83.251.161:8200"
  retry {
    num_retries = 5
  }
}

cache {
  use_auto_auth_token = true
}

auto_auth {
  method "azure" {
    mount_path = "AUTH_MOUNT_PATH"
    namespace = "root"
    config = {
      resource = "https://management.azure.com/"
      role = "VAULT_ROLE"
      client_id = ".....-666f-.....-.....-.........."
    }
  }

  sink {
    type = "file"
    config = {
      path = "/tmp/sink_file.txt"
      mode = 0644
    }
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = true
}

Additional context

hashicorp / vault

Vault Agent doesn't support Azure Container Instances (ACI) with Managed Identities (MSI) #20055