azurerm_linux_function_app: azurerm tries to mutate network although no such change is planned

u3556490 commented 4 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.6.6

AzureRM Provider Version

3.93.0

Affected Resource(s)/Data Source(s)

azurerm_linux_function_app

Terraform Configuration Files

resource "azurerm_linux_function_app" "dap_funcapp" {
  name                = "xxxxxxx"
  resource_group_name = data.azurerm_resource_group.dap_rg.name
  location            = data.azurerm_resource_group.dap_rg.location

  storage_account_name       = data.azurerm_storage_account.dap_storage_acct.name
  storage_account_access_key = data.azurerm_storage_account.dap_storage_acct.primary_access_key
  service_plan_id            = data.azurerm_service_plan.funcapp_service_plan.id

  tags = local.tags

  site_config { }

  app_settings = {
    WEBSITES_ENABLE_APP_SERVICE_STORAGE = true
    WEBSITE_ENABLE_SYNC_UPDATE_SITE     = true
    FUNCTIONS_WORKER_RUNTIME            = "python"
    FUNCTIONS_EXTENSION_VERSION         = "~4"
    SCM_DO_BUILD_DURING_DEPLOYMENT      = false
    AzureWebJobsFeatureFlags            = "EnableWorkerIndexing"
    BUILD_FLAGS                         = "UseExpressBuild"
    ENABLE_ORYX_BUILD                   = "true"
    XDG_CACHE_HOME                      = "/tmp/.cache"
  }

  lifecycle {
    ignore_changes = [
      virtual_network_subnet_id, site_config["ip_restriction"], site_config["vnet_route_all_enabled"]
    ]
  }
}

Debug Output/Panic Output

PUT /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx?api-version=2023-01-01 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 webapps/2023-01-01) HashiCorp Terraform/1.6.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.93.0 VSTS_664bc712-e7c8-4475-8f5b-9a88f9d8df97_build_250_0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 3911
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: redacted
Accept-Encoding: gzip

<request body. note: truncated>
{"id":"/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx","kind":"functionapp,linux","location": ... "virtualNetworkSubnetId":"/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy","vnetContentShareEnabled":false,"vnetImagePullEnabled":false,"vnetRouteAllEnabled":true},"tags": ... }

2024-04-11T11:36:44.437+0800 [DEBUG] provider.terraform-provider-azurerm_v3.93.0_x5.exe: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx?api-version=2023-01-01: timestamp="2024-04-11T11:36:44.423+0800"
2024-04-11T11:36:44.670+0800 [DEBUG] provider.terraform-provider-azurerm_v3.93.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx?api-version=2023-01-01: 
HTTP/2.0 403 Forbidden
Content-Length: 767
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 11 Apr 2024 03:36:44 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: redacted
X-Ms-Failure-Cause: gateway
X-Ms-Request-Id: redacted
X-Ms-Routing-Request-Id: KOREACENTRAL:20240411T033644Z:e4e75263-8d93-43af-9365-37ebc847a9aa
X-Msedge-Ref: Ref A: 8EBCB52744924EC88FD8A1146739450A Ref B: SEL221051504037 Ref C: 2024-04-11T03:36:44Z

{"error":{"code":"LinkedAuthorizationFailed","message":"The client 'redacted' with object id 'redacted' has permission to perform action 'Microsoft.Web/sites/write' on scope '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx'; however, it does not have permission to perform action(s) 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked scope(s) '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy' (respectively) or the linked scope(s) are invalid."}}: timestamp="2024-04-11T11:36:44.661+0800"

Expected Behaviour

The planned change (see actual behaviour for output records) successfully takes place
As the planned change does not contain network changes,
- No change to the network-related attributes of the tfstate
- No change to the actual network

Actual Behaviour

The planned change fails to be deployed.

Full terraform apply output (note some sensitive information is redacted):

data.azurerm_resource_group.dap_rg: Reading...
data.azurerm_resource_group.dap_rg: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/redacted]
data.azurerm_service_plan.funcapp_service_plan: Reading...
data.azurerm_storage_account.dap_storage_acct: Reading...
data.azurerm_service_plan.funcapp_service_plan: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/serverFarms/redacted]
data.azurerm_storage_account.dap_storage_acct: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Storage/storageAccounts/redacted]
azurerm_linux_function_app.dap_funcapp: Refreshing state... [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # azurerm_linux_function_app.dap_funcapp will be updated in-place
  ~ resource "azurerm_linux_function_app" "dap_funcapp" {
      ~ app_settings                                   = {
          + "FUNCTIONS_EXTENSION_VERSION"         = "~4"
            # (8 unchanged elements hidden)
        }
        id                                             = "/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxxx"
        name                                           = "xxxxxxx"
        # (27 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
azurerm_linux_function_app.dap_funcapp: Modifying... [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx]
╷
│ Error: updating Linux App Service (Subscription: "redacted"
│ Resource Group Name: "redacted"
│ Site Name: "xxxxxxx"): performing CreateOrUpdate: unexpected status 403 with error: LinkedAuthorizationFailed: The client 'redacted' with object id 'redacted' has permission to perform action 'Microsoft.Web/sites/write' on scope '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx'; however, it does not have permission to perform action(s) 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked scope(s) '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy' (respectively) or the linked scope(s) are invalid.
│ 
│   with azurerm_linux_function_app.dap_funcapp,
│   on main.tf line 51, in resource "azurerm_linux_function_app" "dap_funcapp":
│   51: resource "azurerm_linux_function_app" "dap_funcapp" {
│ 
│ updating Linux App Service (Subscription:
│ "redacted"
│ Resource Group Name: "redacted"
│ Site Name: "xxxxxxx"): performing CreateOrUpdate: unexpected
│ status 403 with error: LinkedAuthorizationFailed: The client
│ 'redacted' with object id
│ 'redacted' has permission to perform action
│ 'Microsoft.Web/sites/write' on scope
│ '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx';
│ however, it does not have permission to perform action(s)
│ 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked
│ scope(s)
│ '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy'
│ (respectively) or the linked scope(s) are invalid.
╵
##[error]Cmd.exe exited with code '1'.

Steps to Reproduce

The scenario, in general, is as follows:

Terraform is being run on a CI/CD agent
- The agent is Azure Pipelines self-hosted agent on a Windows VM
- Terraform uses the VM's managed identity, which does not have permissions to mutate networks and will not get it
The resource to manage is an Azure Function App
- Created with Terraform
- The Vnet integration is manually done after resource provisioned (CI/CD not applicable)
- Integrated to a subnet, specified in the App Service Plan
After Vnet integrated, changes to the FuncApp are attempted (e.g. app settings / tags changes)
- terraform plan correctly shows the changes as it is in the code
- most importantly, no network changes are involved
- terraform apply still shows the right changes, but fails when applying them

Re-creating the resources from scratch (except the network) does not help.

Using debug output, it was found that the interfering attribute is virtual_network_subnet_id, which has been ignored in the Terraform code. The tfstate keeps this attribute empty; when Terraform tries to connect to the AzureRM API endpoint to create/update the FuncApp, the request body correctly contains the actual non-empty subnet ID (which it obtained from prior calls to check the actual infrastructure state). This is where it fails inexplicably - AzureRM assumes a network change is being made where in reality there is none.

Important Factoids

The identity with which the terraform commands are run does not have permissions to mutate network resources, this should not distract one from the nature of the issue.

References

No response

u3556490 commented 4 months ago

Sorry for bumping this issue but I wonder whether anyone else has encountered similar issues? If this is an issue with the Azure RM SDK or Azure REST API I could go ask them instead. Thanks

u3556490 commented 3 months ago

Update: using an account with identical access levels (same role assigned), it is possible to manually update application settings of a VNET-integrated Function App in the Azure Portal. Just that using Terraform would not work. To rule out the possibility of an upstream problem (prob w/ MS Azure REST API) I will probably go open a ticket with them

gpataneext commented 2 months ago

Hi, I've the same issue. I've a function app with a vnet subnet assigned. I'm trying to change only docker image_name and image_tag in application stack block (inside site_config block), but I get an error which says that I need the permission "Microsoft.Network/virtualNetworks/subnets/join/action" on the linked subnet which is the one assigned to the function app. I did some tests and found that the error is related to the parameter "_virtual_network_subnetid", and I'm sure about that because if I remove the function app subnet assignment, and try to perform the same docker image name and tag change, I can perform the action through the same terraform code. So, why do I need particular role assignments to virtual network to only change the docker image parameters? Is there any ongoing process to solve this bug?

u3556490 commented 2 months ago

Update: I created a ticket to Microsoft support and mentioned Terraform sends a PUT request to an Azure REST API endpoint (as shown in the original post), upon which it apparently fails. They explained it is because API indeed tries to validate the whole request body, i.e. they treat it as the whole configuration and as one object. Their original words:

Yes, setting the virtualNetworkSubnetId site property is the same as configuring VNet integration and requires the same permissions. Then the question is why touching other unrelated config would need permissions of VNet integration.

Thanks a lot for your analysis and sharing the detailed traces. We can see that Terraform uses this API Web Apps - Create Or Update Configuration - REST API (Azure App Service) | Microsoft Learn to update the site configuration (same thing would happen for ARM template, Azure CLI, etc.). The request body contains all of the configuration. This design is to treat site properties as a single object.

Therefore the following 3 permissions would be required for your Terraform credentials when it manages VNet integrated apps.

Action Description Microsoft.Network/virtualNetworks/read Read the virtual network definition Microsoft.Network/virtualNetworks/subnets/read Read a virtual network subnet definition Microsoft.Network/virtualNetworks/subnets/join/action Joins a virtual network

I wonder if there is an alternative implementation for Terraform AzureRM provider that only sends the required changes to Azure REST API - that would be nice.

hashicorp / terraform-provider-azurerm