Adding containers has a permissions issue when running from Github Runner.

[Dipak-Mistry-WTW]

Is there an existing issue for this?

• [X] I have searched the existing issues

Greenfield/Brownfield provisioning

greenfield

Terraform Version

1.81

Module Version

0.1.1

AzureRM Provider Version

3.99.0

Affected Resource(s)/Data Source(s)

azapi_resource

Terraform Configuration Files

module "storage_account_emea" {
  depends_on                    = [azurerm_resource_group.emea]
  source                        = "git::https://github.com/Azure/terraform-azurerm-avm-res-storage-storageaccount.git"
  name                          = lower("*****${local.environment_short}data")
  resource_group_name           = azurerm_resource_group.emea.name
  public_network_access_enabled = true
  is_hns_enabled                = true
  shared_access_key_enabled     = true
  use_nested_nacl               = true
  network_rules = {
    default_action = "Deny"
    bypass         = ["AzureServices"]
    ip_rules       = local.ip_rules
  }
  containers = {
    name = {
      name = "bronze"
    }
  }
  private_endpoints = {
    pe1 = {
      inherit_tags        = true
      location            = azurerm_resource_group.emea.location
      resource_group_name = azurerm_resource_group.emea.name
      subnet_resource_id  = var.peered_vnet_subnet_id
      subresource_name    = ["blob"]
    }
  }

tfvars variables values

Not needed.

Debug Output/Panic Output

╷
│ Error: checking for presence of existing Resource: (ResourceId "/subscriptions/***/resourceGroups/***-******-*-rgrp/providers/Microsoft.Storage/storageAccounts/*****/blobServices/default/containers/bronze" / Api Version "2022-09-01"): ChainedTokenCredential authentication failed
│ GET http://169.254.169.254/metadata/identity/oauth2/token
│ --------------------------------------------------------------------------------
│ RESPONSE 400 Bad Request
│ --------------------------------------------------------------------------------
│ {
│   "error": "invalid_request",
│   "error_description": "Identity not found"
│ }
│ --------------------------------------------------------------------------------
│ 
│

Expected Behaviour

It should create the container resource.

However the OIDC user assigned managed identity doesn't seem to be getting passed through to the azapi_resource type.

Actual Behaviour

If the resource exists it can't plan. If it doesn't exist it plans but doesn't apply. The error is as the logs.

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[matt-FFFFFF]

The provider is trying to auth by MSI.

I got around this by creating an iptables rule to reject all outbound traffic to that IP.

---

• name: iptables block MSI endpoint run: sudo iptables -I OUTPUT --destination 169.254.169.254 -j REJECT

[Dipak-Mistry-WTW]

The provider is trying to auth by MSI.

I got around this by creating an iptables rule to reject all outbound traffic to that IP.

• name: iptables block MSI endpoint run: sudo iptables -I OUTPUT --destination 169.254.169.254 -j REJECT

Thanks @matt-FFFFFF - But in my case I am using a managed identity for auth. If i add that task it presents another error which is:

`##[debug]Error: reading "Resource: (ResourceId \"/subscriptions/*/resourceGroups/**-rgrp/providers/Microsoft.Storage/storageAccounts/*****data/blobServices/default/containers/bronze\" / Api Version \"2022-09-01\")": ChainedTokenCredential: failed to acquire a token.

[debug]Attempted credentials:

[debug] managed identity timed out

[debug] AzureCLICredential: ERROR: Please run 'az login' to setup account.

[debug]

[debug]

[debug] with module.storage_account_emea.azapi_resource.containers["name"],

[debug] on .terraform/modules/storage_account_emea/main.containers.tf line 2, in resource "azapi_resource" "containers":

[debug] 2: resource "azapi_resource" "containers" {

[debug]

[debug]`

[matt-FFFFFF]

Oh ok. That's odd. I use UAMI in my test pipelines and that works well.

Are you using self-hosted runners?

[Dipak-Mistry-WTW]

Not using self hosted. Just using Github hosted.

[matt-FFFFFF]

So you're using OIDC not MSI?

In that case my previous recommendation stands. Reject traffic to the metadata IP and configure the providers to use OIDC as per the docs.

[Dipak-Mistry-WTW]

@matt-FFFFFF Yes using oidc but its still not working.

I still get this error.

Error: reading "Resource: (ResourceId \"/subscriptions/***/resourceGroups/***-***-***/providers/Microsoft.Storage/storageAccounts/********/blobServices/default/containers/applications-state\" / Api Version \"2022-09-01\")": ChainedTokenCredential: failed to acquire a token.

Happy to jump on a call to have a look next week.

My yaml looks like this:

      # See https://github.com/Azure/terraform-azurerm-avm-res-storage-storageaccount/issues/74
      - name: iptables block MSI endpoint
        run: sudo iptables -I OUTPUT --destination 169.254.169.254 -j REJECT

      - name: Terraform Plan
        id: tf-plan
        run: |
          export exitcode=0
          terraform plan -detailed-exitcode -no-color -out ${{ inputs.tfvars-filename }}.tfplan -var-file="${{ inputs.subscription }}/${{ inputs.tfvars-filename }}.tfvars" || export exitcode=$?

          echo "exitcode=$exitcode" >> $GITHUB_OUTPUT

          if [ $exitcode -eq 1 ]; then
            echo Terraform Plan Failed!
            exit 1
          else 
            exit 0
          fi

[matt-FFFFFF]

Hi this isn't an issue with the module, it's the way the provider is trying to authenticate.

GET http://169.254.169.254/metadata/identity/oauth2/token

The request the MSI endpoint is failing.

You either need to block access to this using iptables:

sudo iptables -I OUTPUT --destination 169.254.169.254 -j REJECT

or disable MSI auth in the provider

[ZdenekPesek]

Hello, seems I got into the same troubles.

 Error: Failed to retrieve resource

   with azapi_resource.containers["tfstate"],
   on main.containers.tf line 2, in resource "azapi_resource" "containers":
    2: resource "azapi_resource" "containers" {

 reading Resource: (ResourceId
 "/subscriptions/xxx/resourceGroups/xxx-rg/providers/Microsoft.Storage/storageAccounts/xxx/blobServices/default/containers/tfstate"
 / Api Version "2023-01-01"): ChainedTokenCredential: failed to acquire a
 token.
 Attempted credentials:
    managed identity timed out
    AzureCLICredential: Azure CLI not found on path

I use OIDC on self hosted GH runners with provider configured as

provider "azapi" {
  environment = "public"
  tenant_id = "${local.tenant_id}"
  subscription_id  = "${local.subscription_id}"
  client_id  = "${local.client_id}"
  use_oidc = true
  use_msi  = false
  skip_provider_registration = true
}

But I can't make it working. Any hint would be appreciated.

Thanks

[matt-FFFFFF]

Can you try adding the iptables step?

You might need to use sudo..

RR

[ZdenekPesek]

@matt-FFFFFF all seems to fine after all. No need to create any iptables rule after provider was correctly setup. thanks

[matt-FFFFFF]

To help others can you post your provider config here?

[ZdenekPesek]

It is exactly the same as I have provided in my first comment. Problem was rather with the GH workflow itself. But for the sake of completeness I provide all azure providers together running with OIDC.

generate "azure_provider" {
  path      = "_azure_provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "azurerm" {
  features {}
  environment     = "public"

  tenant_id       = "${local.tenant_id}"
  subscription_id = "${local.subscription_id}"
  client_id       = "${local.client_id}"
  use_oidc        = true

  storage_use_azuread        = true
  skip_provider_registration = true
}

provider "azapi" {
  environment     = "public"

  tenant_id       = "${local.tenant_id}"
  subscription_id = "${local.subscription_id~}"
  client_id       = "${local.client_id}"
  use_oidc        = true
  use_msi         = false

  skip_provider_registration = true
}

provider "azuread" {
  environment     = "public"

  tenant_id       = "${local.tenant_id}"
  client_id       = "${local.client_id}"
  use_oidc        = true
}
EOF
}

generate "provider_versions" {
  path      = "provider_versions_override.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  required_providers {
    azapi = {
      source = "azure/azapi"
    }
  }
}
EOF
}

generate "backend" {
  path      = "_backend.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOT
terraform {
  backend "azurerm" {
    use_azuread_auth = true

    environment     = "public"
    tenant_id       = "${local.tenant_id}"
    subscription_id = "${local.infra_subscription_id}" 
    client_id       = "${local.client_id}"
    use_oidc        = true

    resource_group_name  = "${local.backend_resource_group_name}"
    storage_account_name = "${local.backend_storage_account_name}"
    container_name       = "tfstate"
    key                  = "${path_relative_to_include()}/terraform.tfstate"
  }
}
EOT
}

[Dipak-Mistry-WTW]

Mine is now working since I added:

provider "azapi" { use_oidc = true }

I had the provider for azurerm but was missing the azapi provider with the oidc flag.

[matt-FFFFFF]

Thanks - we will consider this resolved but please let us know if this is not the case #RR

[microsoft-github-policy-service[bot]]

[microsoft-github-policy-service[bot]]

[!IMPORTANT] @Dipak-Mistry-WTW, this issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[!TIP] To prevent further actions to take effect, one of the following conditions must be met:

• The author must respond in a comment within 3 days of this comment.
• The "Status: No Recent Activity :zzz:" label must be removed.
• If applicable, the "Status: Long Term :hourglass_flowing_sand:" or the "Needs: Module Owner :mega:" label must be added.

[microsoft-github-policy-service[bot]]

[!WARNING] @Dipak-Mistry-WTW, this issue will now be closed, as it has been marked as requiring author feedback but has not had any activity for 7 days.

[!TIP] In case this issue needs to be reopened (e.g., the author responds after the issue was closed), the "Status: No Recent Activity :zzz:" label must be removed.