`azurerm_nginx_configuration`: `AnotherOperationInProgress` error while creating configuration or certificate

wuxu92 commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

v1.8.3

AzureRM Provider Version

3.107.0

Affected Resource(s)/Data Source(s)

azurerm_nginx_configuration, azurerm_nginx_certificate

Terraform Configuration Files

TestAccCertificate_update/TestAccConfiguration_basic

Debug Output/Panic Output

------- Stdout: -------
=== RUN   TestAccConfiguration_basic
=== PAUSE TestAccConfiguration_basic
=== CONT  TestAccConfiguration_basic
    testcase.go:113: Step 1/2 error: Error running apply: exit status 1
        Error: creating Configuration (Subscription: "*******"
        Resource Group Name: "acctestRG-auto-240703234536886299"
        Nginx Deployment Name: "acctest-240703234536886299"
        Configuration Name: "default"): polling after ConfigurationsCreateOrUpdate: polling failed: the Azure API returned the following error:
        Status: "Failed"
        Code: "AnotherOperationInProgress"
        Message: "Another operation on this or dependent resource is in progress."
        Activity Id: ""
        ---
        API Response:
        ----[start]----
        {"id":"/providers/Nginx.NginxPlus/locations/EASTUS2/operationStatuses/5d94121b-1b45-4499-85ed-5cb28c3c774c*921CCC2563CE66BBAB76ED7B3EC329516B03A6F75623B89A673CA3EEC1432656","name":"5d94121b-1b45-4499-85ed-5cb28c3c774c*921CCC2563CE66BBAB76ED7B3EC329516B03A6F75623B89A673CA3EEC1432656","resourceId":"/subscriptions/*******/resourceGroups/acctestRG-auto-240703234536886299/providers/Nginx.NginxPlus/nginxDeployments/acctest-240703234536886299/configurations/default","status":"Failed","startTime":"2024-07-03T23:48:55.5574818Z","endTime":"2024-07-03T23:48:55.8398362Z","error":{"code":"AnotherOperationInProgress","message":"Another operation on this or dependent resource is in progress."}}
        -----[end]-----
          with azurerm_nginx_configuration.test,
          on terraform_plugin_test.tf line 125, in resource "azurerm_nginx_configuration" "test":
         125: resource "azurerm_nginx_configuration" "test" {
        creating Configuration (Subscription: "*******"
        Resource Group Name: "acctestRG-auto-240703234536886299"
        Nginx Deployment Name: "acctest-240703234536886299"
        Configuration Name: "default"): polling after ConfigurationsCreateOrUpdate:
        polling failed: the Azure API returned the following error:
        Status: "Failed"
        Code: "AnotherOperationInProgress"
        Message: "Another operation on this or dependent resource is in progress."
        Activity Id: ""
        ---
        API Response:
        ----[start]----
        {"id":"/providers/Nginx.NginxPlus/locations/EASTUS2/operationStatuses/5d94121b-1b45-4499-85ed-5cb28c3c774c*921CCC2563CE66BBAB76ED7B3EC329516B03A6F75623B89A673CA3EEC1432656","name":"5d94121b-1b45-4499-85ed-5cb28c3c774c*921CCC2563CE66BBAB76ED7B3EC329516B03A6F75623B89A673CA3EEC1432656","resourceId":"/subscriptions/*******/resourceGroups/acctestRG-auto-240703234536886299/providers/Nginx.NginxPlus/nginxDeployments/acctest-240703234536886299/configurations/default","status":"Failed","startTime":"2024-07-03T23:48:55.5574818Z","endTime":"2024-07-03T23:48:55.8398362Z","error":{"code":"AnotherOperationInProgress","message":"Another
        operation on this or dependent resource is in progress."}}
        -----[end]-----
--- FAIL: TestAccConfiguration_basic (434.37s)
FAIL

Expected Behaviour

A lock should be added to sequence the operations on the same Nginx deployment.

Actual Behaviour

    Message: "Another operation on this or dependent resource is in progress."

Steps to Reproduce

No response

Important Factoids

No response

References

No response

wuxu92 commented 1 month ago

@puneetsarna Could you take a look at this Nginx conflict issue?

tombuildsstuff commented 1 month ago

@wuxu92 if the API is returning this then we'll need a lock within the Resource to account for this, fwiw?

wuxu92 commented 1 month ago

@tombuildsstuff You are right! I shared this issue with @puneetsarna before. they are going to look into it to address it either with a lock in azurerm or fix it in the server side eliminating the need for a lock in the provider.

SangeetaCarter commented 1 month ago

@valyria257

valyria257 commented 1 month ago

@wuxu92 is this a regularly occurring failure? I've tried reproducing it locally, but I'm unable to.

valyria257 commented 1 month ago

I found the above example deployment, acctest-240703234536886299, in our logs. It looks like we received an Azure Marketplace webhook in the same second as a PUT request to create the NGINX configuration. Order of events:

deployment became ready
deployment PATCH (marketplace webhook)
configuration PUT - 409 conflict

This seems like a one off, but I'm curious @wuxu92 how often are you seeing this?

wuxu92 commented 1 month ago

@valyria257 This happens occasionally, I guess it is less than 1/10 in the nginx basic acc test. I'm wondering if the marketplace webhook is configured by the tenant or by the Nginx service? as I cannot see any related configuration from my side.

valyria257 commented 1 month ago

@wuxu92 Our Microsoft partner team handles the webhooks, so you're right that it's unrelated to the user's configuration. If you have any other example failures, or run into future ones, can you copy the details here? I want to make sure these failures have the same root cause.

wuxu92 commented 1 month ago

@valyria257 This has occurred 2 or 3 times in the past month, with details identical to this issue.

valyria257 commented 1 month ago

Thanks @wuxu92, could you copy the test output for those failures? It contains the nginx deployment information that I can use to search through our internal logs.

wuxu92 commented 3 weeks ago

@valyria257 got a new AnotherOperationInProgress failure last week:

        Error: creating Nginx Deployment (Subscription: "*******"
        Resource Group Name: "acctestRG-auto-240808001130629913"
        Nginx Deployment Name: "acctest-240808001130629913"): polling after DeploymentsCreateOrUpdate: polling failed: the Azure API returned the following error:
        Status: "Failed"
        Code: "AnotherOperationInProgress"
        Message: "Another operation on this or dependent resource is in progress."
        Activity Id: ""
        ---
        API Response:
        ----[start]----
        {"id":"/providers/Nginx.NginxPlus/locations/EASTUS2/operationStatuses/2f311127-0fec-40c3-9705-411b91e6da19*EF803AD73E0248A034EEEA00C1EC9628B05A2A45680445108CA13852213B4FB8","name":"2f311127-0fec-40c3-9705-411b91e6da19*EF803AD73E0248A034EEEA00C1EC9628B05A2A45680445108CA13852213B4FB8","resourceId":"/subscriptions/*******/resourceGroups/acctestRG-auto-240808001130629913/providers/Nginx.NginxPlus/nginxDeployments/acctest-240808001130629913","status":"Failed","startTime":"2024-08-08T00:12:28.1508864Z","endTime":"2024-08-08T00:13:55.2210789Z","error":{"code":"AnotherOperationInProgress","message":"Another operation on this or dependent resource is in progress."}}
        -----[end]-----
          with azurerm_nginx_deployment.test,
          on terraform_plugin_test.tf line 110, in resource "azurerm_nginx_deployment" "test":
         110: resource "azurerm_nginx_deployment" "test" {
        creating Nginx Deployment (Subscription:
        "*******"
        Resource Group Name: "acctestRG-auto-240808001130629913"
        Nginx Deployment Name: "acctest-240808001130629913"): polling after
        DeploymentsCreateOrUpdate: polling failed: the Azure API returned the
        following error:
        Status: "Failed"
        Code: "AnotherOperationInProgress"
        Message: "Another operation on this or dependent resource is in progress."
        Activity Id: ""
        ---
        API Response:
        ----[start]----
        {"id":"/providers/Nginx.NginxPlus/locations/EASTUS2/operationStatuses/2f311127-0fec-40c3-9705-411b91e6da19*EF803AD73E0248A034EEEA00C1EC9628B05A2A45680445108CA13852213B4FB8","name":"2f311127-0fec-40c3-9705-411b91e6da19*EF803AD73E0248A034EEEA00C1EC9628B05A2A45680445108CA13852213B4FB8","resourceId":"/subscriptions/*******/resourceGroups/acctestRG-auto-240808001130629913/providers/Nginx.NginxPlus/nginxDeployments/acctest-240808001130629913","status":"Failed","startTime":"2024-08-08T00:12:28.1508864Z","endTime":"2024-08-08T00:13:55.2210789Z","error":{"code":"AnotherOperationInProgress","message":"Another
        operation on this or dependent resource is in progress."}}
        -----[end]-----

valyria257 commented 3 weeks ago

Thanks @wuxu92! I'm currently looking into this now.

hashicorp / terraform-provider-azurerm