[FEATURE REQ] promoting operation-level policies in SOAP APIs

[AmitAnajwala]

Please describe the feature.

Hi, When we extract an API having policy at operation level, it creates a folder with OperationId. It should create folder with Operation name.

Issue: I extracted API having operation level policy using extractor pipeline. Then I deleted that API from API Management. I tried to create API with publisher pipeline in same API Management instance.

I got below error: System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/nn/resourceGroups/rg-001/providers/Microsoft.ApiManagement/service/apim-dev-int001-pro/apis/FC-AdminFeeService-Interface/operations/64f70d4e46346101d4e97723/policies/policy?api-version=2022-04-01-preview&format=rawxml failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.

Let me know if any additional information is required.

Thanks, Amit Anajwala.

[github-actions[bot]]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

Again seems like you need to override using the configuration file.

[AmitAnajwala]

Hi Wael, I tried to override by adding configuration.dev.yaml file with following content and referred configuration.dev.yaml file from publisher pipeline.

apimServiceName: apis:

• name: proxy-dp-thdochandler-soap

If I delete the proxy-dp-thdochandler-soap from above APIM instance and then try to create it using publisher pipeline, I am getting below error: '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.

Once API is re-created, operations will have new operation ids. And it won’t be able to apply Operation policy with old operation id.

[waelkdouh]

@guythetechie thoughts on this?

[guythetechie]

We use IDs instead of display names to better align with the APIM REST API. The REST API expects IDs in the URL for APIs, operations, backends, etc.

Older versions of ApiOps used names, but it created all sorts of issues. For example:

• Every time we read a folder name, we had to look up what ID corresponded to the folder name before running operations on it. This made the code much more complex and brittle.
• Many customers had characters in display names that were not supported folder names. For instance, d#s?382<> is a perfectly fine display name, but you cannot create a folder with that name.

[guythetechie]

Once API is re-created, operations will have new operation ids. And it won’t be able to apply Operation policy with old operation id.

Why will it have new operation ids? If you're just publishing what was extracted, the publisher should publish using the original operation IDs.

1. Run extractor. You now have the following structure in Git:
   • artifacts/apis/myapi/operations/operationId1/policy.xml: policy file for operationId1
   • artifacts/apis/myapi/specification.xxx: specification file that contains operationId1
2. Delete myapi from APIM instance in Azure portal.
3. Run publisher. It should recreate the API, its operations (including operationId1), and a policy for operationId1 based on the files extracted in step 1.

[AmitAnajwala]

Thanks for the reply.

If I have policy at "all operations" level, it works fine. If I have a policy at operation level, then only I am getting error.

1. Run extractor. You now have the following structure in Git:
   • artifacts/apis/myapi/operations/operationId1/policy.xml: policy file for operationId1
   • artifacts/apis/myapi/specification.xxx: specification file that contains operationId1
2. Delete myapi from APIM instance in Azure portal.
3. Run publisher. It should recreate the API, its operations (including operationId1), and a policy for operationId1 based on the files extracted in step 1.

If I follow above steps, then it is re-creating the deleted myapi, but it is failing with below error, when trying to apply operationId1 level policy. {"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.

I followed above steps because we have only one APIM instance in Dev.

If we try to publish to APIM instance in higher environment like SIT, then also I am anticipating same error, because when it will create myapi in SIT and create operations, those operation ids will be different from operation ids in dev (where I extracted APIs) and it would fail with Entity with specified identifier not found error.

Can you please confirm if it will work fine when we publish API having operation level policy in higher environment?

Or please let me know how to override it in higher environment using configuration file.

Regards, Amit.

From: guythetechie @.> Sent: Monday, October 16, 2023 7:39 PM To: Azure/apiops @.> Cc: Amit Jitendrabhai Anajwala @.>; Author @.> Subject: Re: [Azure/apiops] [FEATURE REQ] (Issue #399)

Caution This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Once API is re-created, operations will have new operation ids. And it won't be able to apply Operation policy with old operation id.

Why will it have new operation ids? If you're just publishing what was extracted, the publisher should publish using the original operation IDs.

1. Run extractor. You now have the following structure in Git:

   • artifacts/apis/myapi/operations/operationId1/policy.xml: policy file for operationId1
   • artifacts/apis/myapi/specification.xxx: specification file that contains operationId1

2. Delete myapi from APIM instance in Azure portal.

3. Run publisher. It should recreate the API, its operations (including operationId1), and a policy for operationId1 based on the files extracted in step 1.

- Reply to this email directly, view it on GitHubhttps://github.com/Azure/apiops/issues/399#issuecomment-1764565212, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2RTCRIN72VUZTMXCFRZIHLX7U5YHAVCNFSM6AAAAAA566MUEOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRUGU3DKMRRGI. You are receiving this because you authored the thread.Message ID: @.**@.>>

DISCLAIMER: This communication contains information which is confidential and may also be privileged. It is for the exclusive use of the addressee. If you are not the addressee please note that any distribution, reproduction, copying, publication or use of this communication or the information is prohibited. If you have received this communication in error, please contact us immediately and also delete the communication from your computer. We accept no liability for any loss or damage suffered by any person arising from use of this email.

[TKProlifics]

I work with Amit and I would like to ask a related question.

I guessed the reasons (both of them) for using ids in the operation folder names. However, it seems to me that APIOps is limited to working with a single instance of APIM. We would like to do this:

• Extract the APIs from our DEV APIM instance and commit them to our Git repo
• Use APIOps for future changes in the DEV environment
• Promote our API definitions in the next environment by publishing the same definitions from the same repo into a new (empty) APIM instance.

I do understand that this was not the main goal of the APIOps tool. APIOps is a great time-saver for many teams. However, promoting code between environments is a really important activity and we are almost completely blocked on this currently.

Can the APIOps team offer any advice on how to promote an entire set of APIs, Backends, Named Values etc into a new empty APIM instance? We really liked the idea of using APIOps, and we thought we had a way forward until we hit the issue with the operation-level policies.

If the answer is 'feel free to submit a pull request' then we might be open to that, but only if it can done in a way that is consistent with the project roadmap.

[waelkdouh]

I work with Amit and I would like to ask a related question.

I guessed the reasons (both of them) for using ids in the operation folder names. However, it seems to me that APIOps is limited to working with a single instance of APIM. We would like to do this:

• Extract the APIs from our DEV APIM instance and commit them to our Git repo

• Use APIOps for future changes in the DEV environment

• Promote our API definitions in the next environment by publishing the same definitions from the same repo into a new (empty) APIM instance.

I do understand that this was not the main goal of the APIOps tool. APIOps is a great time-saver for many teams. However, promoting code between environments is a really important activity and we are almost completely blocked on this currently.

Can the APIOps team offer any advice on how to promote an entire set of APIs, Backends, Named Values etc into a new empty APIM instance? We really liked the idea of using APIOps, and we thought we had a way forward until we hit the issue with the operation-level policies.

If the answer is 'feel free to submit a pull request' then we might be open to that, but only if it can done in a way that is consistent with the project roadmap.

The intent of APIops is to actually allow you to promote across different APIM instances. We offer a configuration system to override values that you would like to change in the target APIM instance (e.g different backend url for an operation). Seems to me that our tool already offers what you're asking for here.

[guythetechie]

@AmitAnajwala - can you zip and attach here the contents of the artifacts/apis folder after running the extractor? Please scrub any sensitive information. You can also create a private GitHub repo with your artifacts and invite me to it.

[TKProlifics]

@guythetechie: Thanks for the very prompt response.

It's good to hear that we're not so far off the map as I feared. However, I still don't know how to use APIOps to do this:

• Create an API which has multiple operations, each operation having its own set of policies
• Publish that API into an APIM instance that has never hosted any version of that API

Right now we have a set of 23 APIs each with up to 4 operations and the only way to promote them into a new environment is to manually copy each operation's policy XML via the APIM portal. This is time-consuming and error-prone, as I'm sure you can imagine.

Can you advise?

[guythetechie]

@TKProlifics - I assume you're creating your initial APIs using the Azure portal. Here is the happy path I would expect:

1. You create your APIs in the dev portal, assign policies to operations, etc.
2. Once you're happy with the dev state, you run the extractor.
3. The extractor generates this file structure:
   • artifacts/apis/api1/apiInformation.json
   • artifacts/apis/api1/specification.xxx. This specification file should contain all your operations, including the APIM-generated operation IDs. For REST APIs, the operation ID should be fined in paths.somePath.httpMethod.operationId.

     
     openapi: 3.0.1
     info:
     title: Swagger API Team2
     description: A sample API that uses a petstore as an example to demonstrate features in the OpenAPI 3.0 specification
     termsOfService: http://swagger.io/terms/
     contact:
     name: Swagger API Team
     url: http://swagger.io/
     email: apiteam@swagger.io
     license:
     name: Apache 2.0
     url: https://www.apache.org/licenses/LICENSE-2.0.html
     version: '1.0'
     servers:

   • url: https://apiopskdalfddev.azure-api.net/3223 paths: /pets: get: summary: findPets description: "Returns all pets from the system that the user has access to\nNam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis. Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum libero. Morbi euismod sagittis libero sed lacinia.\n\nSed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique, sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien.\n" operationId: findPets0 // Here is your operation ID

   • artifacts/apis/api1/policy.xml. This should be your API-level policy, if you use one.
   • artifacts/apis/api1/operations/operationId1/policy.xml. This policy is scoped to operationId1. Again, operationId1 should be defined in your extracted specification.xxx.
4. You run the publisher on your empty PROD instance (having updated your configuration accordingly).
5. The publisher creates the extracted APIs with the operation IDs specified in your specification file. It then assigns operation policies.

I can think of two reasons why things wouldn't work this way:

• A bug with the publisher (certainly possible).
• The operation IDs in your folder path don't match the IDs in the specification file.

Seeing the files generated by the extractor would help me narrow down the issue.

[AmitAnajwala]

Hi @waelkdouh and @guythetechie, I am attaching APIs extracted using APIOps extractor pipeline after removing sensitive information from it. APIs_Extract.zip

Thanks, Amit.

[AmitAnajwala]

@TKProlifics - I assume you're creating your initial APIs using the Azure portal. Here is the happy path I would expect:

1. You create your APIs in the dev portal, assign policies to operations, etc.
2. Once you're happy with the dev state, you run the extractor.
3. The extractor generates this file structure:

• artifacts/apis/api1/apiInformation.json
• artifacts/apis/api1/specification.xxx. This specification file should contain all your operations, including the APIM-generated operation IDs. For REST APIs, the operation ID should be fined in paths.somePath.httpMethod.operationId.

openapi: 3.0.1
info:
  title: Swagger API Team2
  description: A sample API that uses a petstore as an example to demonstrate features in the OpenAPI 3.0 specification
  termsOfService: http://swagger.io/terms/
  contact:
    name: Swagger API Team
    url: http://swagger.io/
    email: apiteam@swagger.io
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
  version: '1.0'
servers:
  - url: https://apiopskdalfddev.azure-api.net/3223
paths:
  /pets:
    get:
      summary: findPets
      description: "Returns all pets from the system that the user has access to\nNam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis. Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum libero. Morbi euismod sagittis libero sed lacinia.\n\nSed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique, sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien.\n"
      operationId: findPets0 // Here is your operation ID

• artifacts/apis/api1/policy.xml. This should be your API-level policy, if you use one.
• artifacts/apis/api1/operations/operationId1/policy.xml. This policy is scoped to operationId1. Again, operationId1 should be defined in your extracted specification.xxx.

4. You run the publisher on your empty PROD instance (having updated your configuration accordingly).
5. The publisher creates the extracted APIs with the operation IDs specified in your specification file. It then assigns operation policies.

I can think of two reasons why things wouldn't work this way:

• A bug with the publisher (certainly possible).
• The operation IDs in your folder path don't match the IDs in the specification file.

Seeing the files generated by the extractor would help me narrow down the issue.

Hi, I think issue is due to second reason in above reply. I checked extracted specification.wsdl file and operation ids are not present in that file.

If I manually export the API with OpenAPI v3(YAML) or OpenAPI v3(JSON) format, then Operation Ids are present in those files.

Regards, Amit,

[guythetechie]

@AmitAnajwala - you're correct. APIM doesn't export operation IDs in WSDL specification files (unlike OpenAPI specification files).

Since the specification files have no operation IDs, APIM generates new ones when the publisher runs. These don't match the original IDs, so it fails when applying operation-level policies.

I guess you found an unsupported ApiOps scenario: promoting operation-level policies in SOAP APIs. I don't have any clean solutions off the top of my head, unfortunately. It would work if you converted to OpenAPI specs, but I certainly understand why you would want to keep using SOAP.

[TKProlifics]

Thanks @guythetechie - that makes sense, although it sounds like bad news for our project. You're correct that we have no choice about using SOAP/WSDL APIs.

If I understand correctly, the set of data written by the exporter does not contain enough information to recreate the original APIs. So this is not (only) a limitation in the publisher. The publisher uses the exported operation ids (held as folder names in the filesystem), but there is no way to link those operation ids to the correct operation names in the WSDL specification.

Questions:

• Is there any way to resolve this by changing only the publisher? I don't think there is but I just want to check my understanding.
• If the exporter was modified to put the operation name somewhere in the operation folder, would that offer a way forward? I'm thinking that the publisher could then query the newly-minted API and find out its new operation ids. Then the publisher could use the operation names in the exported data to find the correct operation id for each operation. Not sure whether that would introduce new sequencing requirements in the publisher though.
• Is there any value in using operation names as folder names when exporting WSDL APIs? I know there were good reasons for moving away from this approach. Just wondering whether a limited use of this idea could offer a solution for this problem?

[guythetechie]

If I understand correctly, the set of data written by the exporter does not contain enough information to recreate the original APIs. So this is not (only) a limitation in the publisher. The publisher uses the exported operation ids (held as folder names in the filesystem), but there is no way to link those operation ids to the correct operation names in the WSDL specification.

That's correct. Using operation names in the folder will not help you though, because the REST API calls rely on IDs. To create an operation policy, you make PUT request to https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ApiManagement/service/{serviceName}/apis/{apiId}/operations/{operationId}/policies/policy?api-version=2022-08-01. Note that the URL expects the operation ID, not its name. Any solution will have to include the ID.

Your second bullet point might be viable, except that we really want to make sure that IDs are aligned across environments. This is a key tenet for many things to work, including configuration overrides. We could make an exception for SOAP operations, but things will get messy...

[guythetechie]

I also noticed that APIM allows you to export SOAP APIs into OpenAPI specification files, which will include the correct operation IDs. Not sure if that helps, but good to know.

[TKProlifics]

Summary of what we know so far...

• APIOps assumes that the API specification file will contain operation ids.
• Extract / publish of an API using OpenAPIv3 preserves the operation ids.
• Extract / publish of an API using WSDL does not preserve the operation ids unless the API already exists in the target APIM instance.

Because of the above, APIOps cannot promote a SOAP API into a new instance of APIM if it has operation-level policies.

The operation ids are considered an integral part of the API specification and must be preserved when an API is extracted/re-published. This rules out any solution that allows APIOps to cope with environment-specific operation ids (so no point in pursuing the idea of using operation names as folder names).

We need a short-term workaround for our project. I wonder whether this would work:

• Export SOAP APIs as OpenAPIv3 and store the exported YAML in our repo alongside the extracted API definitions.
• Import the API from the OpenAPI spec before publishing the API using APIOps. This will definitely create an API with the correct operation ids. Hopefully it will also preserve the essential SOAP-ness of the API and will not interfere with the APIOps publish (which will attempt to import the API from the WSDL).

We will test out this approach, but we would appreciate any advice/feedback on potential issues.

[mirzos]

I am having the same issue. Looks like I will have to do the workaround which Devs may not appreciate :(

[mmmbai]

This is very critical feature. Half of our APIs are SOAP based and have operation level policy, Template, Query, Headers, Request and Response. Without this feature we can not use APIOPS tools to extract/publish to another environment. @waelkdouh @guythetechie Do you have any plan to implement this feature?