search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.6k stars 4.64k forks source link

Unable to create azurerm_api_management_api with imported content file with azurerm > 3.69.0 #23322

Open mennlo opened 1 year ago

mennlo commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

1.5.7

AzureRM Provider Version

3.69.0

Affected Resource(s)/Data Source(s)

azurerm_api_management_api

Terraform Configuration Files

data "azurerm_api_management" "apim" {
  name                = var.apim_name
  resource_group_name = var.resource_group_name
}

resource "azurerm_api_management_api" "api" {
  for_each = var.managed_apis

  name                  = var.instance_id != null ? "${each.key}-${var.instance_id}" : each.key
  display_name          = "${each.value.display_name} [${upper(var.env)}]"
  resource_group_name   = data.azurerm_api_management.apim.resource_group_name
  api_management_name   = data.azurerm_api_management.apim.name
  revision              = "current"
  soap_pass_through     = each.value.import_type == "wsdl" ? true : false
  path                  = local.is_shared_service ? "${local.app_context}/${each.value.service_path}" : each.value.service_path
  protocols             = ["https"]
  subscription_required = each.value.subscription_required
  version               = each.value.version != null ? each.value.version : null
  version_set_id        = each.value.version_set != null ? azurerm_api_management_api_version_set.version_set[each.value.version_set].id : null

  dynamic "import" {
    for_each = each.value.api_file == null ? [] : [1]
    content {
      content_format = each.value.import_type
      content_value  = file("${var.paths.apis}/${each.value.api_file}")

      dynamic "wsdl_selector" {
        for_each = each.value.wsdl_selector

        content {
          service_name  = wsdl_selector.value.service_name
          endpoint_name = wsdl_selector.value.endpoint_name
        }
      }
    }
  }
  depends_on = [azurerm_api_management_api_version_set.version_set]
}

EXAMPLE managed_apis var:

managed_apis = {
  "testapi" = {
    display_name = "test"
    service_path = "test"
    import_type  = "openapi"
    // This product id will have to match one of the the product id's in the 
    // Azure Portal in the APIs -> Products section of the APIM.  If you don't
    // see one that matches the project your API is for then reach out your
    // team lead (if not using a shared services apim) or the Cloud Services
    // team in MS Teams (if using share services apim)
    product_id            = "test"
    api_file              = "test.openapi.yaml"
    subscription_required = false
    wsdl_selector         = []
  },
}

EXAMPLE test.openapi.yaml

openapi: 3.0.0
info:
  title: title
  description: endpoint
  version: 1.0.0
servers:
  - url: "https://redacted.com/test/v1/test/test"
    description: environment
paths:
  /default:
    post:
      operationId: default
      summary: Default
      description: Default operation
      responses:
        "200":
          description: Accepted
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/response"
components:
  schemas:
    response:
      type: object
      properties:
        status:
          type: string
          example: success
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic

Debug Output/Panic Output

terraform [init -upgrade=false]
  Running command terraform with args [init -upgrade=false]

  Initializing the backend...

  Initializing provider plugins...
  - Finding hashicorp/azurerm versions matching "< 4.0.0"...
  - Installing hashicorp/azurerm v3.73.0...
  - Installed hashicorp/azurerm v3.73.0 (signed by HashiCorp)

  Terraform has created a lock file .terraform.lock.hcl to record the provider
  selections it made above. Include this file in your version control repository
  so that Terraform can guarantee to make the same selections by default when
  you run "terraform init" in the future.

  Terraform has been successfully initialized!

  You may now begin working with Terraform. Try running "terraform plan" to see
  any changes that are required for your infrastructure. All Terraform commands
  should now work.

  If you ever set or change modules or backend configuration for Terraform,
  rerun this command to reinitialize your working directory. If you forget, other
  commands will detect it and remind you to do so if necessary.
  terraform [apply -input=false -auto-approve -var managed_apis={"test-api-v1" = {"display_name" = "Test API", "service_path" = "test/test", "subscription_required" = false, "wsdl_selector" = [], "product_id" = "test", "api_file" = "test.openapi.yaml", "import_type" = "openapi", "version" = null, "version_set" = null}} -var project_context=testing -var version_sets=[] -var instance_id=256 -var env=dev -var api_operations=[] -var apim_product={} -var api_policies=[] -var operation_policies=[] -var named_values={} -var resource_group_name=rg-redacted-01 -var apim_name=apim-redacted-01 -var paths={"apis" = "/home/runner/work/terraform-azure-managed-api-module/terraform-azure-managed-api-module/test/apis", "policies" = null} -no-color -lock=false]
  Running command terraform with args [apply -input=false -auto-approve -var managed_apis={"test-api-v1" = {"display_name" = "Test API", "service_path" = "test/test", "subscription_required" = false, "wsdl_selector" = [], "product_id" = "test", "api_file" = "test.openapi.yaml", "import_type" = "openapi", "version" = null, "version_set" = null}} -var project_context=testing -var version_sets=[] -var instance_id=256 -var env=dev -var api_operations=[] -var apim_product={} -var api_policies=[] -var operation_policies=[] -var named_values={} -var resource_group_name=rg-redacted-01 -var apim_name=apim-redacted-01 -var paths={"apis" = "/home/runner/work/terraform-azure-managed-api-module/terraform-azure-managed-api-module/test/apis", "policies" = null} -no-color -lock=false]
  data.azurerm_api_management.apim: Reading...
  data.azurerm_api_management.apim: Read complete after 1s [id=/subscriptions/***/resourceGroups/rg-redacted-01/providers/Microsoft.ApiManagement/service/apim-redacted-01]

  Terraform used the selected providers to generate the following execution
  plan. Resource actions are indicated with the following symbols:
    + create

  Terraform will perform the following actions:

    # azurerm_api_management_api.api["test-api-v1"] will be created
    + resource "azurerm_api_management_api" "api" {
        + api_management_name   = "apim-redacted-01"
        + api_type              = (known after apply)
        + display_name          = "Test API [DEV]"
        + id                    = (known after apply)
        + is_current            = (known after apply)
        + is_online             = (known after apply)
        + name                  = "test-api"
        + path                  = "test/test"
        + protocols             = [
            + "https",
          ]
        + resource_group_name   = "rg-redacted-01"
        + revision              = "current"
        + service_url           = (known after apply)
        + soap_pass_through     = false
        + subscription_required = false
        + version               = (known after apply)
        + version_set_id        = (known after apply)

        + import {
            + content_format = "openapi"
            + content_value  = <<-EOT
                  openapi: 3.0.0
                  info:
                    title: title
                    description: endpoint
                    version: 1.0.0
                  servers:
                    - url: "https://redacted.com/test/v1/test/test"
                      description: sandbox environment
                  paths:
                    /default:
                      post:
                        operationId: default
                        summary: Default
                        description: Default operation
                        responses:
                          "200":
                            description: Accepted
                            content:
                              application/json:
                                schema:
                                  $ref: "#/components/schemas/response"
                  components:
                    schemas:
                      response:
                        type: object
                        properties:
                          status:
                            type: string
                            example: success
                    securitySchemes:
                      basicAuth:
                        type: http
                        scheme: basic
              EOT
          }
      }

    # azurerm_api_management_product_api.product_api["test-api-v1"] will be created
    + resource "azurerm_api_management_product_api" "product_api" {
        + api_management_name = "apim-redacted-01"
        + api_name            = "test-api"
        + id                  = (known after apply)
        + product_id          = "test"
        + resource_group_name = "rg-redacted-01"
      }

  Plan: 2 to add, 0 to change, 0 to destroy.
  azurerm_api_management_api.api["test-api-v1"]: Creating...
  azurerm_api_management_api.api["test-api-v1"]: Still creating... [10s elapsed]

  Warning: Argument is deprecated

    with azurerm_api_management_api.api["test-api-v1"],
    on main.tf line 14, in resource "azurerm_api_management_api" "api":
    14:   soap_pass_through     = each.value.import_type == "wsdl" ? true : false

  `soap_pass_through` will be removed in favour of the property `api_type` in
  version 4.0 of the AzureRM Provider

  (and one more similar warning elsewhere)

  Error: creating/updating Api (Subscription: "***"
  Resource Group Name: "rg-redacted-01"
  Service Name: "apim-redacted-01"
  Api: "test-api"): polling after CreateOrUpdate: executing request: unexpected status 404 with error: ResourceNotFound: Api not found.

    with azurerm_api_management_api.api["test-api-v1"],
    on main.tf line 6, in resource "azurerm_api_management_api" "api":

Expected Behaviour

The azurerm_api_management_api should be created successfully.

I'm trying to create an azurerm_api_management_api using an openapi type imported file.

Actual Behaviour

With any version > 3.69.0, The azurerm_api_management_api creation fails with the following error:

  Error: creating/updating Api (Subscription: "***"
  Resource Group Name: "rg-redacted-01"
  Service Name: "apim-dedacted-01"
  Api: "test-api"): polling after CreateOrUpdate: executing request: unexpected status 404 with error: ResourceNotFound: Api not found.

If you used <= 3.69.0 the creation is successful.

Steps to Reproduce

terraform apply

Important Factoids

No response

References

No response

crizzs commented 1 year ago
Screenshot 2023-09-20 at 2 57 21 PM

Im facing this issue too. Whenever we tried to apply changes to the apim, the outcomes are unpredictable as well.

rcskosir commented 1 year ago

@mennlo Thanks for taking the time to open this issue. Please subscribe to PR #23348 that @sinbai has created.

zhallgato commented 1 year ago

It looks like the PR didn't solve the problem. It still occurs to me on v3.75.0.

Can we reopen this issue or should I make a new one?

mennlo commented 1 year ago

It looks like the PR didn't solve the problem. It still occurs to me on v3.75.0.

Can we reopen this issue or should I make a new one?

It worked for me after using v3.75.0.

DominikKrissVisma commented 1 year ago

I also have this issue, also when using AzureRM provider in version 3.75.0 (mbfrahry FYI)

rcskosir commented 1 year ago

@zhallgato, @mennlo, @DominikKrissVisma Thank you for the update, I will reopen this issue.

rcskosir commented 1 year ago

Hey @sinbai, mind taking another look at this one?

sinbai commented 1 year ago

Hi @DominikKrissVisma, @zhallgato , sorry that PR could not solve your problem. Since the PR already addressed mennlo's issue , I assume that your case might be different. Is it possible to provide detailed terraform configuration (easily reproducible and contains variable values - except sensitive information) and reproduce steps to help reproduce and resolve the problem? In addition, it might help a lot if Terraform trace logs could also be provided. Thanks in advance.

DominikKrissVisma commented 1 year ago

hello, this is terraform file (ranamed from tf to txt): genericapi.txt notice we are using azurerm_api_management_api_version_set, and revision in order to force API recreation. The reason is that when API yaml file changes, like there is new endpoint added, those changes are not always deployed via terraform. When this happens we are expiriencing the 404 issue:

Api: "int-c-client-systems-v2"): polling after CreateOrUpdate: executing request: unexpected status 404 with error: ResourceNotFound: Api not found.

  with azurerm_api_management_api.generic_client_systems["ccc"],
  on api_management_generic.tf line 31, in resource "azurerm_api_management_api" "generic_client_systems":
  31: resource "azurerm_api_management_api" "generic_client_systems"

the issue is that API is actualy created during that run, but not tracked in the TF state file, therefore after this happes we need to import the API into state file.

hieumoscow commented 12 months ago

This issue still persists in 3.80, I had to downgrade to 3.68 for the resource to create. Any timeline to fix this issue?

christianfosli commented 11 months ago

I tried to work-around this by downgrading from 3.80 to 3.68 but it seems the state is no longer compatible, so it's not the easiest work-around.

I get a bunch of

│ The current state of azurerm_api_management_product_policy.<policyName>
│ was created by a newer provider version than is currently selected. Upgrade
│ the azurerm provider to work with this state.

│ The current state of azurerm_api_management_api_policy.<policyName>
│ was created by a newer provider version than is currently selected. Upgrade
│ the azurerm provider to work with this state.

Sorry for the noise :-)

tombuildsstuff commented 11 months ago

@sinbai looks like the original PR didn't resolve this issue - since you sent the original PR to fix this, can you take another look into this one? Thanks!

sinbai commented 11 months ago

Can anyone here provide more contexts in terms of step-by-step-repro-this-issue?

christianfosli commented 11 months ago

Can anyone here provide more contexts in terms of step-by-step-repro-this-issue?

I can probably make one, but I can't make the time the next couple of days.

I can start by sharing some more information about the resource were this issue occurred to me on azurerm 3.80, though.

The resource looks like this

resource "azurerm_api_management_api" "api" {
  name                  = "clientname-${var.env}-api-${var.api_name}" # e.g. "clientname-d-api-clientapi-internal"
  api_management_name   = var.api_management_name         # e.g. "clientname-d-apim-common"
  resource_group_name   = var.resource_group_name         # e.g. "clientname-d-rg-common"
  revision              = var.revision                    # e.g. "50923", "50930", ...
  display_name          = var.display_name                # e.g. "clientname clientapi internal dev"
  path                  = "${var.api_name}"               # e.g. "clientapi"
  protocols             = ["https"]
  subscription_required = var.subscription_required       # e.g. true
  service_url           = var.service_url                 # e.g. https://clientname-d-aks.norwayeast.cloudapp.azure.com/clientapi
  import {
    content_format = "openapi"
    content_value  = file(var.openapi_definition)         # a fairly ordinary OpenAPI v3 spec in json format
                                                          # with info.version set to "internal"
  }
  lifecycle {
    # The version number is pulled from the BUILD_ID, so it changes constantly.
    # This will only update if the actual API has changed.
    ignore_changes = [revision]
  }
}

We have just one azurerm_api_management_api resource with this name+api_management_name, so the "revision" property is just being used to track which CI pipeline run it came from.

zhallgato commented 11 months ago

I have some new information.

We had 6 APIMs in different environments (4 of them in East US and 2 of them in East US 2 Azure regions).

We only experienced the above-mentioned issue on those APIMs which were in East US 2 region.

So, we recreated these two APIMs in East US, and since the recreation we no longer experience the issue. Everything works fine.

Maybe the issue is related only to specific Azure regions. (maybe there are some delays in these regions?)

cbruce80 commented 11 months ago

This is happening to me too. We are in US East 2 as well and running azurerm 3.80. We are also importing a fairly large openapi definition.

kjetiloen commented 11 months ago

For us, it seems to happen when trying to add apis with "large" (openapi) definition files (+300KB, +90 operations). Smaller definition files worked fine, and reverting to 3.69.0 works for everything. Location Norway East.

sinbai commented 11 months ago

I really appreciate everyone's help. It has been reproduced and is under investigation. Thank you so much!

celsocoutinho-tangany commented 11 months ago

I am hitting this issue as well, on 3.81.0. My openai definition is quite small, having only 3 operations

sinbai commented 11 months ago

base-layer: the PollUntilDone of CreateOrUpdateThenPoll stops polling when the API returns resource not found error on first poll hashicorp/go-azure-sdk

Hi @tombuildsstuff I assume that this is a base layer issue. For the case of long running operation poller and the API return Location refers to itself, the 404 errors should be additionally handled in the base layer, what do you think? Please refer to issue #740 for details.

tgolly commented 11 months ago

Just started hitting this as well. AzureRM version 3.71, we're only adding one API with a single operation.

If downgrading isn't an option due to state file compatibility, are there any other workarounds besides create manually and import?

arsubram-akamai commented 10 months ago

Any update on this issue? We also get this issue with 3.79.0 & 3.85.0. Our yaml file size is 7.4K.

Also if we just rerun the terraform all works fine. Is there any other work around for this?

mrickly commented 9 months ago

@sinbai : We have observed something similar. I have created a small reproducer based on one of our real life deployments. I can consistently reproduce the failure/success of an api creation by making a small change to the openapi.yaml descriptor. Given the following parameter in the unique REST resource of our openapi.yaml file, the creation of the corresponding api will fail. 

        - description: status
          in: query
          name: status
          schema:
            type: string
            default: P
            enum:
              - PRODUCTIVE
              - ARCHIVED

A summary of the failure pattern is 

GET versionset -> 404
PUT versionset -> 201
GET versionset -> 200
GET version -> 404
PUT version -> 202
GET version -> 404

If we modify the default to be default: PRODUCTIVE however, the creation succeeds, with the following success pattern: 

GET versionset -> 404
PUT versionset -> 201
GET versionset -> 200
GET version -> 404
PUT version -> 202
GET version -> 200
...

Note that https://apitools.dev/swagger-parser/online/ for instance validates both syntaxes. The Azure portal on the other hand doesn't: If we try to create the API by uploading the file via frontend using default: P, we get the following error message: One or more fields contain incorrect values: Default value must be present in the list of values. Our hypothesis is that the version creation fails silently in Azure API management.

We are currently on version v3.85.0

mrickly commented 9 months ago

@sinbai : Do you plan to take a look at this issue again (see my comment above)? Do you want me to open a new issue?

celsocoutinho-tangany commented 9 months ago

@kjetiloen @hieumoscow since this will probably take months to address, can you please share the configuration you used to provision on 3.68.0 or 3.69.0? I am having the same error in those versions...

eugeneromero commented 8 months ago

Latest azurerm provider (version 3.92) still exhibits this behavior. Any progress information on this issue? :)

sinbai commented 8 months ago

@sinbai : Do you plan to take a look at this issue again (see my comment above)? Do you want me to open a new issue?

Hi @mrickly the best solution to this issue is to modify the implementation of base layer SDK. Could you please track this SDK issue for more updates?

sinbai commented 8 months ago

Latest azurerm provider (version 3.92) still exhibits this behavior. Any progress information on this issue? :)

Hi @eugeneromero Could you please track https://github.com/hashicorp/go-azure-sdk/issues/740 SDK issue for more updates?

mrickly commented 8 months ago

@sinbai : Do you plan to take a look at this issue again (see my comment above)? Do you want me to open a new issue?

Hi @mrickly the best solution to this issue is to modify the implementation of base layer SDK. Could you please track this SDK issue for more updates?

Hi @sinbai : I will do that, although I'm not sure that it covers our case. Unless the azure api systematically returns a delayed response via PollUntilDone (possibly different than 404 and 201, 422 for instance), I would expect to get a validation error already in the CreateOrUpdate step.

nikhil0499 commented 8 months ago

Hi Team, Is there any fixes going on ? Please let me know the solution. I am facing the similar issue, like while I am trying to deploying a new API.

Regards, Nikhil

Ayzrian commented 8 months ago

Make sure that the name value of the azurerm_api_management_api resource does not contain any / characters. What happens is that the the azurerm would include the name into the API call it makes, so if your name is test/test this will result into PUT /subscriptions/$SUB_ID/resourceGroups/$RG_NAME/providers/Microsoft.ApiManagement/service/$APIM_NAME/apis/test/test;rev=1?api-version=2022-08-01z

And this is incorrect API request to Azure API, so it will result into 404 :)

Ayzrian commented 8 months ago

Also try running TF_LOG=debug terraform apply. You will be able to see the API requests the provider does, and then troubleshoot it from there. At least it helped me.

celsocoutinho-tangany commented 8 months ago

Can you please share a working configuration @Ayzrian ?

ltmleo commented 7 months ago

I had the same problem using version 3.96.0 downgrading to 3.46.0 works fine.

The logs TF_LOG=debug terraform apply wasn't much helpful, just show the following behavior:

...
module.api.azurerm_api_management_api.api: Creating...
GET /subscriptions/SUBSCRIPTION/resourceGroups/RG/providers/Microsoft.ApiManagement/service/APIM/apis/API;rev=1?api-version=2022-08-01 HTTP/1.1
HTTP/2.0 404 Not Found
{"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}: timestamp=2024-03-21T08:39:38.727-0300
2024-03-21T08:39:38.727-0300 [DEBUG] provider.terraform-provider-azurerm_v3.96.0_x5: Importing API Management API "APIf;rev=1" of type "openapi": timestamp=2024-03-21T08:39:38.727-0300
PUT /subscriptions/SUBSCRIPTION/resourceGroups/RG/providers/Microsoft.ApiManagement/service/APIM/apis/API;rev=1?api-version=2022-08-01 HTTP/1.1
HTTP/2.0 202 Accepted
module.api.azurerm_api_management_api.api: Still creating... [10s elapsed]
2024-03-21T08:39:49.185-0300 [DEBUG] provider.terraform-provider-azurerm_v3.96.0_x5: AzureRM Request
GET /subscriptions/SUBSCRIPTION/resourceGroups/RG/providers/Microsoft.ApiManagement/service/APIM/apis/API;rev=1?api-version=2022-08-01 HTTP/1.1
HTTP/2.0 404 Not Found
{"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}: timestamp=2024-03-21T08:39:49.740-0300
2024-03-21T08:39:49.742-0300 [ERROR] provider.terraform-provider-azurerm_v3.96.0_x5: Response contains error diagnostic: diagnostic_severity=ERROR tf_provider_addr=provider tf_rpc=ApplyResourceChange tf_proto_version=5.4 @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_detail= tf_resource_type=azurerm_api_management_api diagnostic_summary="creating/updating Api (Subscription: "SUBSCRIPTION"
Api: "API;rev=1"): polling after CreateOrUpdate: executing request: unexpected status 404 with error: ResourceNotFound: Api not found." tf_req_id=req_id timestamp=2024-03-21T08:39:49.740-0300
... (some logs showing some operations in the backend)
2024-03-21T08:39:50.290-0300 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-21T08:39:50.297-0300 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.96.0/darwin_arm64/terraform-provider-azurerm_v3.96.0_x5 pid=70672
2024-03-21T08:39:50.297-0300 [DEBUG] provider: plugin exited
cwiederspan commented 7 months ago

I am having this same problem and tried it with the new 3.97.1 version of the AzureRM and am still getting the same error. I had hoped that it was fixed per some of the comments last week, but seems like it's still there.

jpatton-wildfork commented 7 months ago

I'm also getting this issue on provider version 3.97.1.

thomasschaller1 commented 7 months ago

I'm also getting this issue on provider version 3.97.1.

Same (and as well with 3.92.0)

lewis-mtf commented 7 months ago

I am currently getting this on provier version 3.98.0

deankelly780 commented 7 months ago

I have just moved from 3.89 to 3.98 in an attempt to fix this issue, but to no avail:

Error: creating/updating Api (Subscription: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" │ Resource Group Name: "rg-01-aaaf-cs-eastus-stg" │ Service Name: "apim-02-aaaf-cs-eastus-stg" │ Api: "api-fa-001-aaaf-cs-eastus-stg;rev=1"): polling after CreateOrUpdate: executing request: unexpected status 404 (404 Not Found) with error: ResourceNotFound: Api not found. │ │ with module.mod_aaaf_funcapp_001.azurerm_api_management_api.aaaf_apim_api_funcapp_001, │ on modules\100_funcapps\funcapp_001\main.tf line 123, in resource "azurerm_api_management_api" "aaaf_apim_api_funcapp_001": │ 123: resource "azurerm_api_management_api" "aaaf_apim_api_funcapp_001"

I'd really appreciate a fix or a work-around, please. I have literally hundreds of Function Apps to deploy into APIM

sharatbhaskar1988 commented 7 months ago

@deankelly780 did you get any solution for this? The strange part is when i run the terraform apply again then it works in second time.

deankelly780 commented 7 months ago

No @sharatbhaskar1988 nothing yet. I am still working on it. If I come up with something, I'll ping it here.

The failure pattern seems different for me: I am currently developing the TF by deploying 2 x APIs and one works every time, but the other fails every time.

manicminer commented 7 months ago

@sinbai Are you able to take another look at this issue? Thanks!

deankelly780 commented 7 months ago

@manicminer and @sharatbhaskar1988 - I seem to have a solution that at least worked for my particular version of this issue.

When I was deploying, I was using two copies of a mostly identical template to create two different APIs. The only differences were the names of the Function Apps.

Both templates were using the same setting for "path":

resource "azurerm_api_management_api" "aaaf_apim_api_funcapp_001" {
  name                              = "api-${var.aaaf_funcapp_001_name}"
  resource_group_name               = var.aaaf_resource_group_name
  api_management_name               = var.aaaf_apim_name
  revision                          = "1"
  display_name                      = var.aaaf_funcapp_001_name
  path                              = "public"
  protocols                         = [ "https" ]
  import {
    content_format                  = "openapi"
    content_value                   = file("modules/100_funcapps/funcapp_001/openapi_spec.yaml")
  }
    depends_on                      = [ azurerm_api_management_backend.aaaf_apim_backend_funcapp_001 ]
}

By ensuring that "path" was set differently in both templates the problem went away. I hope this is useful to you.

sinbai commented 7 months ago

@sinbai Are you able to take another look at this issue? Thanks!

@manicminer thanks for your PR. Due to testing locally with the following TF configuration, the reproducible situation mentioned in issue #740 (This error occurs when the API returns a 404 error during the first polling. If the polling API returns 200 for the first time, the problem could not be reproduced) did not occur (After trying several times, the API did not return 404.), I'm not sure why the issue still occurs after applying the PR.

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.98.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "exampleRG"
  location = "westeurope"
}

resource "azurerm_api_management" "test" {
  name                = "example"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  publisher_name      = "pub1"
  publisher_email     = "pub1@email.com"

  sku_name = "Consumption_0"
}

resource "azurerm_api_management_api" "test" {
  name                = "example"
  resource_group_name = azurerm_resource_group.test.name
  api_management_name = azurerm_api_management.test.name
  display_name        = "api1"
  path                = "api1"
  protocols           = ["https"]
  # once created, update revision to 3, then run terraform apply
  revision            = "1"

  import {
    content_value  = file("api_management_api_swagger.json")
    content_format = "swagger-json"
  }
}

However, it could be seen from the following log provided by ltmleo that TF polls the Get request after the PUT request, the API returns a 404 record only once. If the log is generated by applying the PR and no additional 404 records are missed, I assume that the retry 404 code in the PR does not seem to be working.

...
module.api.azurerm_api_management_api.api: Creating...
GET /subscriptions/SUBSCRIPTION/resourceGroups/RG/providers/Microsoft.ApiManagement/service/APIM/apis/API;rev=1?api-version=2022-08-01 HTTP/1.1
HTTP/2.0 404 Not Found
{"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}: timestamp=2024-03-21T08:39:38.727-0300
2024-03-21T08:39:38.727-0300 [DEBUG] provider.terraform-provider-azurerm_v3.96.0_x5: Importing API Management API "APIf;rev=1" of type "openapi": timestamp=2024-03-21T08:39:38.727-0300
PUT /subscriptions/SUBSCRIPTION/resourceGroups/RG/providers/Microsoft.ApiManagement/service/APIM/apis/API;rev=1?api-version=2022-08-01 HTTP/1.1
HTTP/2.0 202 Accepted
module.api.azurerm_api_management_api.api: Still creating... [10s elapsed]
2024-03-21T08:39:49.185-0300 [DEBUG] provider.terraform-provider-azurerm_v3.96.0_x5: AzureRM Request
GET /subscriptions/SUBSCRIPTION/resourceGroups/RG/providers/Microsoft.ApiManagement/service/APIM/apis/API;rev=1?api-version=2022-08-01 HTTP/1.1
HTTP/2.0 404 Not Found
{"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}: timestamp=2024-03-21T08:39:49.740-0300
2024-03-21T08:39:49.742-0300 [ERROR] provider.terraform-provider-azurerm_v3.96.0_x5: Response contains error diagnostic: diagnostic_severity=ERROR tf_provider_addr=provider tf_rpc=ApplyResourceChange tf_proto_version=5.4 @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_detail= tf_resource_type=azurerm_api_management_api diagnostic_summary="creating/updating Api (Subscription: "SUBSCRIPTION"
Api: "API;rev=1"): polling after CreateOrUpdate: executing request: unexpected status 404 with error: ResourceNotFound: Api not found." tf_req_id=req_id timestamp=2024-03-21T08:39:49.740-0300
... (some logs showing some operations in the backend)
2024-03-21T08:39:50.290-0300 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-21T08:39:50.297-0300 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.96.0/darwin_arm64/terraform-provider-azurerm_v3.96.0_x5 pid=70672
2024-03-21T08:39:50.297-0300 [DEBUG] provider: plugin exited

Regardless, since the issue could not be reproduced, the root cause could not be located. If anyone here could provide the minimum reproducible TF configuration, repro steps and TF Log information, it would be of great help for us to troubleshoot.

mrickly commented 6 months ago

@sinbai : When an invalid open api file is used, terraform 3.98 still fails with 404.

openapi:

openapi: 3.0.1
info:
  title: j7t-reproducer-service
  version: "1"
servers:
- url: rest
  variables: {}
tags:
- name: tag1
  description: description of tag1
paths:
  /document:
    get:
      description: description of operation1
      operationId: operation1
      parameters:
        - description: status
          in: query
          name: status
          schema:
            type: string
            default: P
            enum:
              - PRODUCTIVE
              - ARCHIVED
      responses:
        "200":
          description: get doc by status

last couple of lines in the terraform log 

2024-04-11T12:43:10.379Z [DEBUG] provider.terraform-provider-azurerm_v3.98.0_x5: PUT https://management.azure.com/subscriptions/e5e059e9-92ef-4e3c-a352-00554175fc0f/resourceGroups/caz-preprod-rg/providers/Microsoft.ApiManagement/service/caz-preprod-apim/apis/j7t-reproducer-service;rev=1?api-version=2022-08-01: timestamp=2024-04-11T12:43:10.379Z
2024-04-11T12:43:10.640Z [DEBUG] provider.terraform-provider-azurerm_v3.98.0_x5: AzureRM Response for https://management.azure.com/subscriptions/e5e059e9-92ef-4e3c-a352-00554175fc0f/resourceGroups/caz-preprod-rg/providers/Microsoft.ApiManagement/service/caz-preprod-apim/apis/j7t-reproducer-service;rev=1?api-version=2022-08-01: 
HTTP/2.0 202 Accepted
Content-Length: 0
Cache-Control: no-cache
Date: Thu, 11 Apr 2024 12:43:09 GMT
Expires: -1
Location: https://management.azure.com/subscriptions/e5e059e9-92ef-4e3c-a352-00554175fc0f/resourceGroups/caz-preprod-rg/providers/Microsoft.ApiManagement/service/caz-preprod-apim/apis/j7t-reproducer-service;rev=1?api-version=2022-08-01&asyncId=6617dadefb4268092441a3a0&asyncCode=201
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 55645181-aba8-cf02-d26f-00b83707aed0
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1199
X-Ms-Request-Id: 55645181-aba8-cf02-d26f-00b83707aed0
X-Ms-Routing-Request-Id: WESTEUROPE:20240411T124310Z:30e04330-c4bc-4619-95ef-31c5ab5b3de5: timestamp=2024-04-11T12:43:10.640Z
2024-04-11T12:43:20.650Z [DEBUG] provider.terraform-provider-azurerm_v3.98.0_x5: AzureRM Request: 
GET /subscriptions/e5e059e9-92ef-4e3c-a352-00554175fc0f/resourceGroups/caz-preprod-rg/providers/Microsoft.ApiManagement/service/caz-preprod-apim/apis/j7t-reproducer-service;rev=1?api-version=2022-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 api/2022-08-01) HashiCorp Terraform/1.7.5 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.98.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Accept: application/json; charset=utf-8; IEEE754Compatible=false
Content-Type: application/json; charset=utf-8
Odata-Maxversion: 4.0
Odata-Version: 4.0
X-Ms-Correlation-Request-Id: 55645181-aba8-cf02-d26f-00b83707aed0
Accept-Encoding: gzip: timestamp=2024-04-11T12:43:20.650Z
2024-04-11T12:43:20.650Z [DEBUG] provider.terraform-provider-azurerm_v3.98.0_x5: GET https://management.azure.com/subscriptions/e5e059e9-92ef-4e3c-a352-00554175fc0f/resourceGroups/caz-preprod-rg/providers/Microsoft.ApiManagement/service/caz-preprod-apim/apis/j7t-reproducer-service;rev=1?api-version=2022-08-01: timestamp=2024-04-11T12:43:20.650Z
2024-04-11T12:43:20.772Z [DEBUG] provider.terraform-provider-azurerm_v3.98.0_x5: AzureRM Response for https://management.azure.com/subscriptions/e5e059e9-92ef-4e3c-a352-00554175fc0f/resourceGroups/caz-preprod-rg/providers/Microsoft.ApiManagement/service/caz-preprod-apim/apis/j7t-reproducer-service;rev=1?api-version=2022-08-01: 
HTTP/2.0 404 Not Found
Content-Length: 79
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 11 Apr 2024 12:43:20 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 55645181-aba8-cf02-d26f-00b83707aed0
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11998
X-Ms-Request-Id: 55645181-aba8-cf02-d26f-00b83707aed0
X-Ms-Routing-Request-Id: WESTEUROPE:20240411T124320Z:3240c800-8f52-4b70-bfb1-80aaf242554d

{"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}: timestamp=2024-04-11T12:43:20.772Z
2024-04-11T12:43:20.772Z [TRACE] provider.terraform-provider-azurerm_v3.98.0_x5: Called downstream: @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:910 tf_provider_addr=provider tf_req_id=b00d4c5c-fc8c-1784-f3ad-c7fd00990078 tf_resource_type=azurerm_api_management_api @module=sdk.helper_schema tf_rpc=ApplyResourceChange timestamp=2024-04-11T12:43:20.772Z
2024-04-11T12:43:20.773Z [TRACE] provider.terraform-provider-azurerm_v3.98.0_x5: Received downstream response: tf_resource_type=azurerm_api_management_api tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/tf5serverlogging/downstream_request.go:40 tf_req_duration_ms=10611 diagnostic_error_count=1 tf_provider_addr=provider tf_req_id=b00d4c5c-fc8c-1784-f3ad-c7fd00990078 @module=sdk.proto diagnostic_warning_count=0 tf_proto_version=5.4 timestamp=2024-04-11T12:43:20.772Z
2024-04-11T12:43:20.773Z [ERROR] provider.terraform-provider-azurerm_v3.98.0_x5: Response contains error diagnostic: @module=sdk.proto diagnostic_detail=""
  diagnostic_summary=
  | creating/updating Api (Subscription: "e5e059e9-92ef-4e3c-a352-00554175fc0f"
  | Resource Group Name: "caz-preprod-rg"
  | Service Name: "caz-preprod-apim"
  | Api: "j7t-reproducer-service;rev=1"): polling after CreateOrUpdate: executing request: unexpected status 404 (404 Not Found) with error: ResourceNotFound: Api not found.
   tf_proto_version=5.4 tf_provider_addr=provider @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_severity=ERROR tf_rpc=ApplyResourceChange tf_req_id=b00d4c5c-fc8c-1784-f3ad-c7fd00990078 tf_resource_type=azurerm_api_management_api timestamp=2024-04-11T12:43:20.773Z
2024-04-11T12:43:20.773Z [TRACE] provider.terraform-provider-azurerm_v3.98.0_x5: Served request: @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/tf5server/server.go:872 @module=sdk.proto tf_proto_version=5.4 tf_provider_addr=provider tf_req_id=b00d4c5c-fc8c-1784-f3ad-c7fd00990078 tf_rpc=ApplyResourceChange tf_resource_type=azurerm_api_management_api timestamp=2024-04-11T12:43:20.773Z
2024-04-11T12:43:20.845Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-04-11T12:43:20.846Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-04-11T12:43:20.847Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/null/3.2.2/linux_amd64/terraform-provider-null_v3.2.2_x5 pid=357
2024-04-11T12:43:20.847Z [DEBUG] provider: plugin exited
2024-04-11T12:43:20.856Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.98.0/linux_amd64/terraform-provider-azurerm_v3.98.0_x5 pid=366
2024-04-11T12:43:20.856Z [DEBUG] provider: plugin exited

It is not clear to me whether azurerm or some of its libraries actually calls the Location https://management.azure.com/subscriptions/e5e059e9-92ef-4e3c-a352-00554175fc0f/resourceGroups/caz-preprod-rg/providers/Microsoft.ApiManagement/service/caz-preprod-apim/apis/j7t-reproducer-service;rev=1?api-version=2022-08-01&asyncId=6617dadefb4268092441a3a0&asyncCode=201 returned by the PUT (https://learn.microsoft.com/en-us/azure/architecture/patterns/async-request-reply). It does not appear in the log in any case.

sinbai commented 6 months ago

Hi @mrickly thank you very much for the information, I could reproduce your situation. I have filed an Terraform SDK issue to track it. Could you please track it for more updates?

mrickly commented 6 months ago

Thank you @sinbai , I will track the new issue.

sharatbhaskar1988 commented 6 months ago

@manicminer and @sharatbhaskar1988 - I seem to have a solution that at least worked for my particular version of this issue.

When I was deploying, I was using two copies of a mostly identical template to create two different APIs. The only differences were the names of the Function Apps.

Both templates were using the same setting for "path":

resource "azurerm_api_management_api" "aaaf_apim_api_funcapp_001" {
  name                              = "api-${var.aaaf_funcapp_001_name}"
  resource_group_name               = var.aaaf_resource_group_name
  api_management_name               = var.aaaf_apim_name
  revision                          = "1"
  display_name                      = var.aaaf_funcapp_001_name
  path                              = "public"
  protocols                         = [ "https" ]
  import {
    content_format                  = "openapi"
    content_value                   = file("modules/100_funcapps/funcapp_001/openapi_spec.yaml")
  }
    depends_on                      = [ azurerm_api_management_backend.aaaf_apim_backend_funcapp_001 ]
}

By ensuring that "path" was set differently in both templates the problem went away. I hope this is useful to you.

@deankelly780 i have an open api doc which have different paths like /a/b/c and /d/e/f. Not sure what should i set in the path property.

J0F3 commented 6 months ago

@sinbai I just want to let you know that this error can happens during the initial deployment (terraform apply) of an API also. When the API is a bit more complex meaning the api has a lot of resources etc. the creation of the API on the API Management can take some time. In such a case also the wrong polling URL is used and the creation of the API fails also with an 404 error. Even tough the API is actually created on the API Management. Any subsequent deployment will then succeed because the API is actually already created on the API Management.

In the TF_TRACE Log the root cause is good recognizable:

Frist the correct PUT request to create the API is made:

[DEBUG] provider.terraform-provider-azurerm_v3.99.0_x5.exe: PUT https://management.azure.com/subscriptions/1cc5c6b6-15cb-4082-ad3f-38874b1c6488/resourceGroups/caz-test-gaia-switzerlandnorth-rg/providers/Microsoft.ApiManagement/service/jfe-test-apim/apis/jfe-test-api;rev=1?api-version=2022-08-01

For wich the Azure response correctly contents the Location header:

[DEBUG] provider.terraform-provider-azurerm_v3.99.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/1cc5c6b6-15cb-4082-ad3f-38874b1c6488/resourceGroups/caz-test-gaia-switzerlandnorth-rg/providers/Microsoft.ApiManagement/service/jfe-test-apim/apis/jfe-test-api;rev=1?api-version=2022-08-01: 
HTTP/2.0 202 Accepted
Content-Length: 0
Cache-Control: no-cache
Date: Tue, 16 Apr 2024 16:05:59 GMT
Expires: -1
Location: https://management.azure.com/subscriptions/1cc5c6b6-15cb-4082-ad3f-38874b1c6488/resourceGroups/caz-test-gaia-switzerlandnorth-rg/providers/Microsoft.ApiManagement/service/jfe-test-apim/apis/jfe-test-api;rev=1?api-version=2022-08-01&asyncId=661ea1e75ddd170c48700615&asyncCode=201
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 0aff170c-5357-ea12-c025-fe9c194313de
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1199
X-Ms-Request-Id: 0aff170c-5357-ea12-c025-fe9c194313de
X-Ms-Routing-Request-Id: GERMANYWESTCENTRAL:20240416T160559Z:6e839db5-1f26-41f2-8120-635fd7f0594f

But then instead that the URL from the Location header is get polled (according to Track asynchronous Azure operations: url-to-monitor-status) azurerm makes a GET request with the URL of the actual API resource which does not exist yet:

[DEBUG] provider.terraform-provider-azurerm_v3.99.0_x5.exe: AzureRM Request: 
GET /subscriptions/1cc5c6b6-15cb-4082-ad3f-38874b1c6488/resourceGroups/caz-test-gaia-switzerlandnorth-rg/providers/Microsoft.ApiManagement/service/jfe-test-apim/apis/jfe-test-api;rev=1?api-version=2022-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 api/2022-08-01) HashiCorp Terraform/1.8.0 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.99.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Accept: application/json; charset=utf-8; IEEE754Compatible=false
Content-Type: application/json; charset=utf-8
Odata-Maxversion: 4.0
Odata-Version: 4.0
X-Ms-Correlation-Request-Id: 0aff170c-5357-ea12-c025-fe9c194313de
Accept-Encoding: gzip

Consequently Azure response then with 404 which lead then to the failure of the deployment:

[DEBUG] provider.terraform-provider-azurerm_v3.99.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/1cc5c6b6-15cb-4082-ad3f-38874b1c6488/resourceGroups/caz-test-gaia-switzerlandnorth-rg/providers/Microsoft.ApiManagement/service/jfe-test-apim/apis/jfe-test-api;rev=1?api-version=2022-08-01: 
HTTP/2.0 404 Not Found
Content-Length: 79
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Tue, 16 Apr 2024 16:06:09 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 0aff170c-5357-ea12-c025-fe9c194313de
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11999
X-Ms-Request-Id: 0aff170c-5357-ea12-c025-fe9c194313de
X-Ms-Routing-Request-Id: GERMANYWESTCENTRAL:20240416T160610Z:81bc76dc-904c-40ea-b8b5-d333aa296505

{"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}: 
[TRACE] provider.terraform-provider-azurerm_v3.99.0_x5.exe: Called downstream: @module=sdk.helper_schema tf_req_id=5b5f079c-857a-58f5-64c4-58242ddc6e19 tf_resource_type=azurerm_api_management_api @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:910 tf_provider_addr=provider tf_rpc=ApplyResourceChange 
[TRACE] provider.terraform-provider-azurerm_v3.99.0_x5.exe: Received downstream response: tf_resource_type=azurerm_api_management_api tf_rpc=ApplyResourceChange diagnostic_warning_count=0 tf_proto_version=5.4 tf_provider_addr=provider tf_req_duration_ms=12132 tf_req_id=5b5f079c-857a-58f5-64c4-58242ddc6e19 @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/tf5serverlogging/downstream_request.go:40 @module=sdk.proto diagnostic_error_count=1 
[ERROR] provider.terraform-provider-azurerm_v3.99.0_x5.exe: Response contains error diagnostic: @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR
  diagnostic_summary=
  | creating/updating Api (Subscription: "1cc5c6b6-15cb-4082-ad3f-38874b1c6488"
  | Resource Group Name: "caz-test-gaia-switzerlandnorth-rg"
  | Service Name: "jfe-test-apim"
  | Api: "jfe-test-api;rev=1"): polling after CreateOrUpdate: executing request: unexpected status 404 (404 Not Found) with error: ResourceNotFound: Api not found.
   tf_proto_version=5.4 tf_provider_addr=provider @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 tf_req_id=5b5f079c-857a-58f5-64c4-58242ddc6e19 tf_resource_type=azurerm_api_management_api tf_rpc=ApplyResourceChange timestamp="2024-04-16T18:06:10.163+0200"
2024-04-16T18:06:10.163+0200 [TRACE] provider.terraform-provider-azurerm_v3.99.0_x5.exe: Served request: tf_rpc=ApplyResourceChange tf_proto_version=5.4 tf_resource_type=azurerm_api_management_api tf_req_id=5b5f079c-857a-58f5-64c4-58242ddc6e19 @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/tf5server/server.go:872 @module=sdk.proto tf_provider_addr=providert

So I think this will get probably also fixed with https://github.com/hashicorp/go-azure-sdk/issues/962 .

sinbai commented 6 months ago

@J0F3 thank you for providing the above information. Per the TF Log above, I assume that once #962 is resolved, the "Api not found" error would not occur in the above situation. So could you please follow # 962 for more updates?