search

Azure / apiops

APIOps applies the concepts of GitOps and DevOps to API deployment. By using practices from these two methodologies, APIOps can enable everyone involved in the lifecycle of API design, development, and deployment with self-service and automated tools to ensure the quality of the specifications and APIs that they’re building.
https://azure.github.io/apiops
MIT License
320 stars 186 forks source link

[BUG] Publisher fails with Self hosted gateways having API's #425

Closed san360 closed 10 months ago

san360 commented 10 months ago

Release version

APIOps Toolkit for Azure APIM v4.10.3

Describe the bug

The publisher operation fails if APIM is configured with self hosted gateway and have API's published in it. It seems the tool publishes Gateway first and then goes to add API's which are not published at APIM yet.

2023-11-29T15:21:56.0004518Z       Putting gateway AKSSelfHosted...
2023-11-29T15:21:56.5710166Z info: Publisher[0]
2023-11-29T15:21:56.5711131Z       Putting API basic-calculator in gateway AKSSelfHosted...
2023-11-29T15:21:56.7541265Z crit: Publisher[0]
2023-11-29T15:21:56.7545975Z       System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/shared-services/providers/Microsoft.ApiManagement/service/san360-apim-ext/gateways/AKSSelfHosted/apis/basic-calculator?api-version=2022-04-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"aid","message":"API not found"}]}}'.
2023-11-29T15:21:56.7549918Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
2023-11-29T15:21:56.7551266Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
2023-11-29T15:21:56.7552607Z          at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
2023-11-29T15:21:56.7553584Z       --- End of stack trace from previous location ---
2023-11-29T15:21:56.7554718Z          at publisher.GatewayApi.Put(ApiName apiName, GatewayUri gatewayUri, PutRestResource putRestResource, CancellationToken cancellationToken)
2023-11-29T15:21:56.7555846Z          at publisher.GatewayApi.<>c__DisplayClass7_0.<<Put>b__1>d.MoveNext()
2023-11-29T15:21:56.7556837Z       --- End of stack trace from previous location ---
2023-11-29T15:21:56.7557998Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2023-11-29T15:21:56.7558845Z       --- End of stack trace from previous location ---
2023-11-29T15:21:56.7559715Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
2023-11-29T15:21:56.7561904Z          at publisher.GatewayApi.Put(GatewayName gatewayName, IReadOnlyCollection`1 apiNames, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-11-29T15:21:56.7563685Z          at publisher.GatewayApi.<>c__DisplayClass1_0.<<Put>b__0>d.MoveNext()
2023-11-29T15:21:56.7564275Z       --- End of stack trace from previous location ---
2023-11-29T15:21:56.7564858Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2023-11-29T15:21:56.7565517Z       --- End of stack trace from previous location ---
2023-11-29T15:21:56.7566370Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
2023-11-29T15:21:56.7568776Z          at publisher.GatewayApi.Put(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-11-29T15:21:56.7572236Z          at publisher.GatewayApi.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-11-29T15:21:56.7575751Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-11-29T15:21:56.7578165Z          at publisher.Publisher.ProcessCommitIdFilesToPut(IReadOnlyCollection`1 commitIdFilesToPut, CancellationToken cancellationToken)
2023-11-29T15:21:56.7579702Z          at publisher.Publisher.RunWithCommitId(CommitId commitId, CancellationToken cancellationToken)
2023-11-29T15:21:56.7580570Z          at publisher.Publisher.Run(CancellationToken cancellationToken)
2023-11-29T15:21:56.7581197Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
2023-11-29T15:21:56.7581706Z info: Microsoft.Hosting.Lifetime[0]
2023-11-29T15:21:56.7582006Z       Application is shutting down...

The code from Service.cs publish Gateway first and then publishes API, Probably Gateway can be publsihed at the end.

https://github.com/Azure/apiops/blob/2713a1f7c5ff779bfefd7d5365d8b2157fea2209/tools/code/publisher/Service.cs#L43

Expected behavior

The Gateway along with all the API's are moved.

Actual behavior

The gateway is moved but API's are not and application fails

Reproduction Steps

  1. Create APIM with self hosted gateway
  2. Define API's and assign them to self hosted gateway
  3. Run the extractor and publisher
github-actions[bot] commented 10 months ago 
  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.
san360 commented 10 months ago

Thanks to @rbickel for support in troubleshooting and identifying the issue

waelkdouh commented 10 months ago

@san360 Kindly post the solution here for others to benefit from in case they encounter a similar issue in the future.

san360 commented 10 months ago

@waelkdouh we were not able to resolve this issue via any workarounds (The only solution is to have API's published before Gateway is published and as mentioned above move the Gateway publish to the end in the Service.cs file), we took the latest from main branch and executed the code locally, In local execution it did not fail on the gateway part which was failing in the pipeline but it failed on AAD group migration part.