base-layer: the `PollUntilDone` of `CreateOrUpdateThenPoll` stops polling when the API returns resource not found error on first poll

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Service Used

APIM

API Versions Used

2022-08-01

Description

While investigating GH #23322, it was found that the PollUntilDone of CreateOrUpdateThenPoll stops polling when the API returns resource not found error although this is only the first polling. Details are as follows:

Base-layer: after investigation, it was found that the error was caused by code terminating polling.
Steps to reproduce GH #23322:

Create azurerm_api_management_api with the following config.

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.38.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "example"
  location = "eastus2"
}

resource "azurerm_api_management" "test" {
  name                = "example"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  publisher_name      = "pub1"
  publisher_email     = "pub1@email.com"

  sku_name = "Consumption_0"
}

resource "azurerm_api_management_api" "test" {
  name                = "example"
  resource_group_name = azurerm_resource_group.test.name
  api_management_name = azurerm_api_management.test.name
  display_name        = "api1"
  path                = "api1"
  protocols           = ["https"]
  revision            = "1"

  import {
    content_value  = file("testdata/api_management_api_swagger.json")
    content_format = "swagger-json"
  }
}

Update the revision of azurerm_api_management_api to "2", then Terraform Apply.

Actual:  An error occurred:  “polling after CreateOrUpdate: executing request: unexpected status 404 with error: ResourceNotFound: Api not found.”

Expected: The resource `azurerm_api_management_api` is successfully recreated.

I assume that the base layer should handle 404 situations, and for LRO, the API returning 404 when the resource is not completely created is expected behavior.

Note: This error occurs when the API returns a 404 error during the first polling. If the polling API returns 200 for the first time, the problem could not be reproduced.

Terraform Logs for step2:

PUT /subscriptions/xxx/resourceGroups/example/providers/Microsoft.ApiManagement/service/example/apis/example;rev=2?api-version=2022-08-01

{"properties":{"apiType":"http","apiVersion":"v1","apiVersionSetId":"/subscriptions/xxx/resourceGroups/example/providers/Microsoft.ApiManagement/service/example/apiVersionSets/example","format":"swagger-json","path":"api1","type":"http","value":"{\n \"swagger\": \"2.0\",\n \"info\": {\n \"description\": \"This is a simple API\",\n \"version\": \"1.0.2\",\n \"title\": \"tftest2\",\n \"contact\": {\n \"email\": \"you@your-company.com\"\n },\n \"license\": {\n \"name\": \"Apache 2.0\",\n \"url\": \"http://www.apache.org/licenses/LICENSE-2.0.html\"\n }\n },\n \"tags\": [\n {\n \"name\": \"admins\",\n \"description\": \"Secured Admin-only calls\"\n },\n {\n \"name\": \"developers\",\n \"description\": \"Operations available to regular developers\"\n }\n ],\n \"paths\": {\n \"/inventory\": {\n \"get\": {\n \"tags\": [\n \"developers\"\n ],\n \"summary\": \"searches inventory\",\n \"operationId\": \"searchInventory\",\n \"description\": \"By passing in the appropriate options, you can search for\\navailable inventory in the system\\n\",\n \"produces\": [\n \"application/json\"\n ],\n \"parameters\": [\n {\n \"in\": \"query\",\n \"name\": \"searchString\",\n \"description\": \"pass an optional search string for looking up inventory\",\n \"required\": false,\n \"type\": \"string\"\n },\n {\n \"in\": \"query\",\n \"name\": \"skip\",\n \"description\": \"number of records to skip for pagination\",\n \"type\": \"integer\",\n \"format\": \"int32\",\n \"minimum\": 0\n },\n {\n \"in\": \"query\",\n \"name\": \"limit\",\n \"description\": \"maximum number of records to return\",\n \"type\": \"integer\",\n \"format\": \"int32\",\n \"minimum\": 0,\n \"maximum\": 50\n }\n ],\n \"responses\": {\n \"200\": {\n \"description\": \"search results matching criteria\",\n \"schema\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"#/definitions/InventoryItem\"\n }\n }\n },\n \"400\": {\n \"description\": \"bad input parameter\"\n }\n }\n },\n \"post\": {\n \"tags\": [\n \"admins\"\n ],\n \"summary\": \"adds an inventory item\",\n \"operationId\": \"addInventory\",\n \"description\": \"Adds an item to the system\",\n \"consumes\": [\n \"application/json\"\n ],\n \"produces\": [\n \"application/json\"\n ],\n \"parameters\": [\n {\n \"in\": \"body\",\n \"name\": \"inventoryItem\",\n \"description\": \"Inventory item to add\",\n \"schema\": {\n \"$ref\": \"#/definitions/InventoryItem\"\n }\n }\n ],\n \"responses\": {\n \"201\": {\n \"description\": \"item created\"\n },\n \"400\": {\n \"description\": \"invalid input, object invalid\"\n },\n \"409\": {\n \"description\": \"an existing item already exists\"\n }\n }\n }\n }\n },\n \"definitions\": {\n \"InventoryItem\": {\n \"type\": \"object\",\n \"required\": [\n \"id\",\n \"name\",\n \"manufacturer\",\n \"releaseDate\"\n ],\n \"properties\": {\n \"id\": {\n \"type\": \"string\",\n \"format\": \"uuid\",\n \"example\": \"d290f1ee-6c54-4b01-90e6-d701748f0851\"\n },\n \"name\": {\n \"type\": \"string\",\n \"example\": \"Widget Adapter\"\n },\n \"releaseDate\": {\n \"type\": \"string\",\n \"format\": \"int32\",\n \"example\": \"2016-08-29T09:12:33.001Z\"\n },\n \"manufacturer\": {\n \"$ref\": \"#/definitions/Manufacturer\"\n }\n }\n },\n \"Manufacturer\": {\n \"required\": [\n \"name\"\n ],\n \"properties\": {\n \"name\": {\n \"type\": \"string\",\n \"example\": \"ACME Corporation\"\n },\n \"homePage\": {\n \"type\": \"string\",\n \"format\": \"url\",\n \"example\": \"https://www.acme-corp.com\"\n },\n \"phone\": {\n \"type\": \"string\",\n \"example\": \"408-867-5309\"\n }\n }\n }\n },\n \"schemes\": [\n \"https\"\n ]\n}\n"}}

Response:

HTTP/2.0 202 Accepted
Content-Length: 0
Cache-Control: no-cache
Date: Wed, 15 Nov 2023 08:21:18 GMT
Expires: -1
Location: https://management.azure.com/subscriptions/xxx/resourceGroups/example/providers/Microsoft.ApiManagement/service/example/apis/example;rev=2?api-version=2022-08-01&asyncId=65547f7f03ed581a686c497a&asyncCode=201
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 3d0c09f5-8809-e787-88a7-8dc1697b6a91
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1199
X-Ms-Request-Id: 3d0c09f5-8809-e787-88a7-8dc1697b6a91
X-Ms-Routing-Request-Id: SOUTHEASTASIA:20231115T082119Z:f82d825c-5109-4818-a807-2373f61d7a26: timestamp=2023-11-15T16:21:19.525+0800

GET /subscriptions/xxx/resourceGroups/example/providers/Microsoft.ApiManagement/service/example/apis/example;rev=2?api-version=2022-08-01

Response: {"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}

References

No response

Looks like this is also an issue intermittently for Recovery Services:

Resource Creation

Request:

:authority: management.azure.com
:method: PUT
:path: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tombuildsstuff-dev/providers/Microsoft.RecoveryServices/vaults/tombuildsstuff-devvault?api-version=2022-10-01
:scheme: https
content-type: application/json; charset=utf-8
user-agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 vaults/2022-10-01) HashiCorp Terraform/1.6.0-dev (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.92.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
x-ms-correlation-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
content-length: 165
accept-encoding: gzip

{
    "identity": {
        "type": "None",
        "userAssignedIdentities": null
    },
    "location": "southindia",
    "properties": {
        "publicNetworkAccess": "Enabled"
    },
    "sku": {
        "name": "Standard"
    },
    "tags": {}
}

Response:

:status: 201
cache-control: no-cache
pragma: no-cache
content-length: 646
content-type: application/json; charset=utf-8
expires: -1
retry-after: 60
x-ms-ratelimit-remaining-subscription-resource-requests: 207
x-ms-correlation-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
azure-asyncoperation: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tombuildsstuff-dev/providers/Microsoft.RecoveryServices/vaults/tombuildsstuff-devvault/operationStatus/XXX=?api-version=2022-10-01&t=XXX&c=XXX
x-content-type-options: nosniff
x-ms-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
x-ms-client-request-id: 7c8c1f72-8794-425a-b322-850a3684c083
server: Kestrel
x-ms-routing-request-id: SWEDENSOUTH:20240216T100543Z:97774991-4e86-43e8-ac59-91783f189a3f
strict-transport-security: max-age=31536000; includeSubDomains
date: Fri, 16 Feb 2024 10:05:43 GMT

{
    "location": "southindia",
    "name": "tombuildsstuff-devvault",
    "etag": "W/\"datetime'2024-02-16T10%3A05%3A43.5505051Z'\"",
    "tags": {},
    "identity": {
        "type": "None"
    },
    "properties": {
        "provisioningState": "Provisioning",
        "privateEndpointStateForBackup": "None",
        "privateEndpointStateForSiteRecovery": "None",
        "redundancySettings": {
            "standardTierStorageRedundancy": "Invalid",
            "crossRegionRestore": "Disabled"
        },
        "publicNetworkAccess": "Enabled"
    },
    "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tombuildsstuff-dev/providers/Microsoft.RecoveryServices/vaults/tombuildsstuff-devvault",
    "type": "Microsoft.RecoveryServices/vaults",
    "sku": {
        "name": "Standard"
    }
}

LRO

Request:

:authority: management.azure.com
:method: GET
:path: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tombuildsstuff-dev/providers/Microsoft.RecoveryServices/vaults/tombuildsstuff-devvault/operationStatus/XXX?api-version=2022-10-01&t=XXX&c=XXX
:scheme: https
content-type: application/json; charset=utf-8
user-agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 vaults/2022-10-01) HashiCorp Terraform/1.6.0-dev (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.92.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
x-ms-correlation-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
accept-encoding: gzip

Response:

:status: 404
cache-control: no-cache
pragma: no-cache
content-type: application/json; charset=utf-8
expires: -1
x-ms-failure-cause: gateway
x-ms-request-id: ed83dc62-d1f3-4b65-abf7-bcc14e216d17
x-ms-correlation-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
x-ms-routing-request-id: SWEDENSOUTH:20240216T100644Z:ed83dc62-d1f3-4b65-abf7-bcc14e216d17
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff
date: Fri, 16 Feb 2024 10:06:44 GMT
content-length: 247

{
    "error": {
        "code": "ResourceNotFound",
        "message": "The Resource 'Microsoft.RecoveryServices/vaults/tombuildsstuff-devvault' under resource group 'tombuildsstuff-dev' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"
    }
}

Subsequent/Repeated LRO Request

Request:

:authority: management.azure.com
:method: GET
:path: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tombuildsstuff-dev/providers/Microsoft.RecoveryServices/vaults/tombuildsstuff-devvault/operationStatus/XXX=?api-version=2022-10-01&t=XXX&c=XXX
:scheme: https
content-type: application/json; charset=utf-8
user-agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 vaults/2022-10-01) HashiCorp Terraform/1.6.0-dev (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.92.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
x-ms-correlation-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
accept-encoding: gzip

Response:

:status: 200
cache-control: no-cache
pragma: no-cache
content-length: 339
content-type: application/json; charset=utf-8
expires: -1
x-ms-ratelimit-remaining-subscription-reads: 11988
x-content-type-options: nosniff
x-ms-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
x-ms-client-request-id: 374fe4f9-053d-4cac-891f-3dcfbaadbf3d
server: Kestrel
x-ms-correlation-request-id: 401e38ec-8f93-e524-05dd-eebed90d4c34
x-ms-routing-request-id: SWEDENSOUTH:20240216T102330Z:4289173c-2f8f-4ee9-9b01-0b25d553feff
strict-transport-security: max-age=31536000; includeSubDomains
date: Fri, 16 Feb 2024 10:23:30 GMT

{
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tombuildsstuff-dev/providers/Microsoft.RecoveryServices/vaults/tombuildsstuff-devvault/operationStatus/XXX",
  "name": "XXX",
  "status": "Succeeded"
}

hashicorp / go-azure-sdk