search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.52k stars 4.6k forks source link

azurerm_windows_web_app not deploying using managed identity for ACR only #19173

Closed alishawong-credera closed 1 year ago

alishawong-credera commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

1.3.4

AzureRM Provider Version

3.30.0

Affected Resource(s)/Data Source(s)

azurerm_windows_web_app

Terraform Configuration Files

resource "azurerm_windows_web_app" "demowebapp" {
  name                = "demowebapp"
  resource_group_name = "rg-demowebapp"
  location            = "uksouth"
  service_plan_id     = data.azurerm_service_plan.example.id

  virtual_network_subnet_id = data.azurerm_subnet.example.id

  site_config {
    application_stack {
      docker_container_name     = var.image_name
      docker_container_registry = var.repo_url
      docker_container_tag      = var.image_tag
    }
    container_registry_use_managed_identity       = true
    container_registry_managed_identity_client_id = data.azurerm_user_assigned_identity.example.client_id
  }

  identity {
    type         = "SystemAssigned, UserAssigned"
    identity_ids = [data.azurerm_user_assigned_identity.example.id]
  }
}

Debug Output/Panic Output

Original Error: Code="BadRequest" Message="The parameter WindowsFxVersion has an invalid value. Unexpected exception while validating OS and version. Image: *******. Error Response: Unauthorized. Request: https://***/oauth2/token?service=***&scope=repository:******:pull. Response: Unauthorized.

Expected Behaviour

The web app to deploy successfully using the assigned Managed Identity which has the permissions to pull from the (private) Azure Container Registry.

Actual Behaviour

Web app doesn't deploy at all. In order for the above code to deploy we need to pass in this:

    app_settings = {
      DOCKER_REGISTRY_SERVER_USERNAME = var.repo_username
      DOCKER_REGISTRY_SERVER_PASSWORD = var.repo_password
    }

which is necessary for private container registries but as we are using Azure Container Registry we want to be able to authenticate to this repo using a MI and not admin credentials. Is this a bug or just not yet supported?

Steps to Reproduce

No response

Important Factoids

No response

References

No response

xiaxyi commented 1 year ago

Thanks @alishawong-credera for raising this issue, I can create the web app using MSI for ACR. May I know if you assigned the acrPull role to the managed identity?

Below is my config, thought I added the app_setting block, I didn't include the password and username:

resource "azurerm_windows_web_app" "test" {
  name                = "xiaxintestwwa-dockermsi"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  service_plan_id     = azurerm_service_plan.test.id

  site_config {
    application_stack {
      docker_container_name = "xiaxin18/windowsserver"
      docker_container_tag  = "nano1809"
    }
    container_registry_use_managed_identity       = true
    container_registry_managed_identity_client_id = azurerm_user_assigned_identity.test.client_id
  }
   app_settings = {
    "DOCKER_REGISTRY_SERVER_URL"          = "xiaxintestacr.azurecr.io"
    "DOCKER_REGISTRY_SERVER_USERNAME"     = ""
    "DOCKER_REGISTRY_SERVER_PASSWORD"     = ""
  }
}
eehret commented 1 year ago

I can confirm I am experiencing the same issue with version 3.33.0 of the azurerm provider.

Initially I was attempting to create the app service using only a system assigned managed identity, but that did not work.

Then I thought maybe the app service is not doing things in the correct sequence; i.e. not waiting for the system assigned managed identity to be created etc, so to rule that out I actually tried the same approach as the above post: creating a user assigned managed identity and AcrPull role assignment on the ACR before creating the app service, then associating that identity with the app service in both the identity block and the container_registry_managed_identity_client_id attribute, but even that didn't work.

It seems right now it's not possible at all to create a Windows app service hooked up to private container registry and pull the image in one go. The only way I've gotten it to work so far is to create the app service initially without trying to pull private image, then after initial provisioning succeeds wire it up to the private registry & private image. But obviously this is not the way we want to use this Terraform feature.

Edit: Note that this is happening while having the "WEBSITE_PULL_IMAGE_OVER_VNET" app setting configured, as is required in our environment because the ACR is private.

alishawong-credera commented 1 year ago

@xiaxyi @eehret

I managed to get round this by not using user identities and using system identities only with the AcrPull role on the ACR. I did what @eehret did - deployed the web app pulling a public image from MCR, then in the deployment pipeline modifying the web app to point to the ACR. Our ACR is not private though.

eehret commented 1 year ago

@alishawong-credera Thanks. Yes, we could do the same here using a subsequent step in the deployment pipeline, although I wonder if this is working as intended. It seems odd to me that you cannot wire up an ACR using a managed identity right from the start.

eehret commented 1 year ago

@alishawong-credera I have reproduced this issue purely using AzureRM templates. So I don't think it's actually a Terraform problem. I have a case open with Microsoft now for this problem, so hopefully they will determine where the problem is and fix it on their end.

xiaxyi commented 1 year ago

Thanks @eehret for the update, can you share the issue link that you raised to the API side with me?

eehret commented 1 year ago

@xiaxyi

Thanks @eehret for the update, can you share the issue link that you raised to the API side with me?

Which issue link are you referring to?

eehret commented 1 year ago

@alishawong-credera @xiaxyi

Just wanted to let you know that Microsoft suggested a workaround to me today that turned out to work. The workaround is to set the App Setting 'DOCKER_SKIP_IMAGE_VALIDATION' to 'true' before creating the Web App service.

I'm not sure how they expect people to know this, as it didn't seem to be documented anywhere...

Hope it helps you :)

rcskosir commented 1 year ago

Thanks for taking the time to submit this issue. It looks like this has been resolved for @alishawong-credera via by not using user identities and using system identities only with the AcrPull role on the ACR, and for @eehret by setting the App Setting 'DOCKER_SKIP_IMAGE_VALIDATION' to 'true' before creating the Web App service. Since this seems to be more of an issue on the Microsoft side of things with documentation, I am going to mark this issue as closed, with the work arounds noted above.

jeru81 commented 9 months ago

@alishawong-credera @xiaxyi

Just wanted to let you know that Microsoft suggested a workaround to me today that turned out to work. The workaround is to set the App Setting 'DOCKER_SKIP_IMAGE_VALIDATION' to 'true' before creating the Web App service.

I'm not sure how they expect people to know this, as it didn't seem to be documented anywhere...

Hope it helps you :)

I am just wondering how you can set AppSetting before creating the Web App service??

Since I am trying to create a windows service with bicep. And app settings need as a parent the app service. So I still get it not deployed with UserManagedId

eehret commented 9 months ago

Hi,

You can set that app setting at the same time as you create the app service. That's what we have been doing to work around this issue.

Sent from my Bell Samsung device over Canada’s largest network.

From: jeru81 @.> Sent: Wednesday, November 29, 2023 10:11:36 AM To: hashicorp/terraform-provider-azurerm @.> Cc: Eric Ehret @.>; Mention @.> Subject: Re: [hashicorp/terraform-provider-azurerm] azurerm_windows_web_app not deploying using managed identity for ACR only (Issue #19173)

@alishawong-crederahttps://github.com/alishawong-credera @xiaxyihttps://github.com/xiaxyi

Just wanted to let you know that Microsoft suggested a workaround to me today that turned out to work. The workaround is to set the App Setting 'DOCKER_SKIP_IMAGE_VALIDATION' to 'true' before creating the Web App service.

I'm not sure how they expect people to know this, as it didn't seem to be documented anywhere...

Hope it helps you :)

I am just wondering how you can set AppSetting before creating the Web App service??****

— Reply to this email directly, view it on GitHubhttps://github.com/hashicorp/terraform-provider-azurerm/issues/19173#issuecomment-1832085042, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AANQE6RU5NQGTTWNWO33B3LYG5GCRAVCNFSM6AAAAAARZNTXUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSGA4DKMBUGI. You are receiving this because you were mentioned.Message ID: @.***>

jeru81 commented 9 months ago

thx eehret, I got it. The point me helped was to set the appSettings with the 'DOCKER_SKIP_IMAGE_VALIDATION' and afterwards the app_config with the WindowsFxVersion -I had to separate it out of the nested siteConfig property where it was initially. Bicep look a like:

`resource app 'Microsoft.Web/sites@2022-09-01' = { name: '${resourceGroupName}-app' location: location tags: tags kind: 'app,container,windows' identity: { type: 'UserAssigned' userAssignedIdentities: { '${userManagedId.id}' : {} } } properties: { enabled: true serverFarmId: resourceId('Microsoft.Web/serverfarms', appServicePlanApp.name) reserved: false isXenon: true hyperV: true vnetRouteAllEnabled: false vnetImagePullEnabled: false vnetContentShareEnabled: false siteConfig: { acrUseManagedIdentityCreds: true acrUserManagedIdentityID: userManagedId.properties.clientId alwaysOn: true ~windowsFxVersion: 'DOCKER|xxxx.azurecr.io/repo/xxxservice:latest'~ } httpsOnly: true publicNetworkAccess: 'Disabled' } }

resource appSettings 'Microsoft.Web/sites/config@2022-09-01' = { parent: app name: 'appsettings' properties: { DOCKER_SKIP_IMAGE_VALIDATION: 'true' APPINSIGHTS_INSTRUMENTATIONKEY: appInsKey APPLICATIONINSIGHTS_CONNECTION_STRING: appInsConStr ApplicationInsightsAgent_EXTENSION_VERSION: '~2' // DOCKER_ENABLE_CI: 'true' DOCKER_REGISTRY_SERVER_URL: 'uri' DOCKER_REGISTRY_SERVER_USERNAME: 'user' DOCKER_REGISTRY_SERVER_PASSWORD: '' WEBSITES_ENABLE_APP_SERVICE_STORAGE: 'false' } }

resource app_config 'Microsoft.Web/sites/config@2022-09-01' = { parent: app name: 'web' properties: { acrUseManagedIdentityCreds: true alwaysOn: true netFrameworkVersion: 'v4.0' windowsFxVersion: 'DOCKER|xxxx.azurecr.io/repo/xxxservice:latest' use32BitWorkerProcess: true managedPipelineMode: 'Integrated' publicNetworkAccess: 'Disabled' scmIpSecurityRestrictionsUseMain: true http20Enabled: true minTlsVersion: '1.2' scmMinTlsVersion: '1.2' ftpsState: 'Disabled' } }`

github-actions[bot] commented 4 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.