search

Azure / apiops

APIOps applies the concepts of GitOps and DevOps to API deployment. By using practices from these two methodologies, APIOps can enable everyone involved in the lifecycle of API design, development, and deployment with self-service and automated tools to ensure the quality of the specifications and APIs that they’re building.
https://azure.github.io/apiops
MIT License
324 stars 191 forks source link

[BUG] Issues with EventHub Logger configured using System Assigned Managed Identity #344

Open Mohid-A opened 1 year ago

Mohid-A commented 1 year ago

Release version

v4.5.0

Describe the bug

We have configured the event hub logger for apim using the system-assigned identity. While running the Publisher to Dev environment is failing with the below exception on the EH logger. I could not able to find any reference in the documentation around eventhub setup using system identity; the wiki and few other issues reported on logger were talking about the logger configured using the connectionString.

exception log:

System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/**-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/**-int-dv-apim/loggers/apim-dv-eventhub-logger?api-version=2022-04-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"One or more Properties ['{0}'] specified are missing.","message":"64c7d150fb42680fe***"}]}}'.

We noticed when the extractor ran that the associated extracted json for the logger only contains the following loggerInformation.json, but it was missing the other properties within the credentials.

`{
  "properties": {
    "credentials": {
      "name": "i**-mgmt-pr-evh-02"
    },
    "description": "adding a new logger with system assigned managed identity",
    "isBuffered": true,
    "loggerType": "azureEventHub"
  }
}`

With the above-extracted json, we noticed the publisher to dev was failing with the missing connectionString property was missing; since we did not use connectionstring to set up the logger and it was using the system identity. I tried to update the JSON logger as below by adding the endpoint and identityClientID (generated named value for the logger), but it failed with the error mentioned in the exception log above.

loggerInformation_manual_update.json


`{
  "properties": {
    "credentials": {
      "identityClientId": "{{64c7d150fb42680**********}}",
      "name": "***-mgmt-pr-evh-02",
      "endpointAddress": "***-mgmt-pr-evhns.servicebus.windows.net/***-mgmt-pr-evh-02"
    },
    "description": "adding a new logger with system assigned managed identity",
    "isBuffered": true,
    "loggerType": "azureEventHub"
  }
}`

Documentation referred for setting up EH logger using system identity https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials

a

Expected behavior

We expect the publisher to dev environment should not fail while pushing the changes back to Dev environment and eventually to higher environments

Actual behavior

System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/gi-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/gi-int-dv-apim/loggers/apim-dv-eventhub-logger?api-version=2022-04-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"One or more Properties ['{0}'] specified are missing.","message":"64c7d150fb42680fe4*****"}]}}'.

Reproduction Steps

  1. EH logger added using system identity for steps mentioned here https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials
  2. Ran extractor and publisher job. Publisher was using the json file loggerInformation.json, where the publisher fails on missing connectionstring property
  3. Manually updated the loggerInformation.json, see above mentioned block for loggerInformation_manual_update.json, where with this config, publisher has failed with the exception as provided above
guythetechie commented 1 year ago

@Mohid-A - according to the documentation, here's the proper JSON structure for Event Hub with managed identities. You were close, except that identityClientId expects the hard-coded value SystemAssigned; not the actual client ID.

I would have expected the client ID as well and agree it's confusing. Please don't shoot the messenger. :)

Mohid-A commented 1 year ago

@guythetechie - I tried that option still no luck

2023-08-02T14:14:48.1720521Z System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/gi-***-dv-apim-rg/providers/Microsoft.ApiManagement/service/***-dv-apim/loggers/apim-dv-eventhub-logger?api-version=2022-04-01-preview failed with status code 500. Content is '{"error":{"code":"InternalServerError","message":"Request processing failed due to internal error.","details":null}}'.

`{
  "properties": {
    "credentials": {
      "name": "i**-mgmt-pr-evh-02",
      "endpointAddress": "i**-mgmt-pr-evhns.servicebus.windows.net/i**-mgmt-pr-evh-02",
      "identityClientId": "SystemAssigned"
    },
    "description": "adding a new logger with system assigned managed identity",
    "isBuffered": true,
    "loggerType": "azureEventHub",
    "resourceId": "/subscriptions/******-****-4642-a937-*********/resourceGroups/*****-pr-rg/providers/Microsoft.EventHub/namespaces/***-mgmt-pr-evhns/eventhubs/***-mgmt-pr-evh-02"
  }
}`
Mohid-A commented 1 year ago

@waelkdouh @guythetechie any suggestions on the fix when you get a chance?

Mohid-A commented 1 year ago

Hi @guythetechie, we are stuck on this issue trying to figure out the fix. Do you have any thoughts on this? Also can't find any reference in apiops documentation/wiki on EH logger using managed identity. The documentation only includes the explanation of implementation using connectionstring. thank you

guythetechie commented 1 year ago

@Mohid-A - I don't think we've tested this scenario specifically (Event Hubs with managed identities). I'll test tomorrow and respond.

guythetechie commented 1 year ago

I was able to successfully create it with this configuration:

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "adding a new logger with system assigned managed identity",
    "credentials": {
         "endpointAddress": "mynamespacename.servicebus.windows.net", // As opposed to mynamespacename.servicebus.windows.net/myeventhub
         "identityClientId":"SystemAssigned",
         "name":"ehub" //  Gave it the same name as my logger, but I don't think it matters
    }
  }
}

Also note that I'm not passing a resource ID; again, don't know if it makes a difference.

Mohid-A commented 1 year ago

@guythetechie I tried with suggested approach still resulted in failure. I have set the logging levels to DEBUG, still see no information around the root cause of the internal server error.

Here's the updated loggerInformation.json I tried on the publish-to-dev job, also tried removing the logger name and id in the subsequent runs too

{
  "properties": {
    "credentials": {
      "endpointAddress": "ioa-mgmt-pr-evhns.servicebus.windows.net",
      "identityClientId": "SystemAssigned",
      "name": "ioa-mgmt-pr-evh-02"
    },
    "description": "adding a new logger with system assigned managed identity",
    "isBuffered": true,
    "loggerType": "azureEventHub",
  "name": "apim-eventhub-logger",
  "id": "/subscriptions/************/resourceGroups/gi-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/gi-int-dv-apim/loggers/apim-eventhub-logger"
  }
}

Exception logs

2023-08-14T23:43:20.5609290Z dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
2023-08-14T23:43:20.5610083Z       Hosting starting
2023-08-14T23:43:20.6771763Z info: Publisher[0]
2023-08-14T23:43:20.6772684Z       Beginning execution...
2023-08-14T23:43:20.6892784Z info: Publisher[0]
2023-08-14T23:43:20.6893427Z       Getting files from commit ID 98a0138745f6f1ea41a426da6c4c92c9126a124f...
2023-08-14T23:43:20.7315877Z info: Microsoft.Hosting.Lifetime[0]
2023-08-14T23:43:20.7316613Z       Application started. Press Ctrl+C to shut down.
2023-08-14T23:43:20.7322221Z info: Microsoft.Hosting.Lifetime[0]
2023-08-14T23:43:20.7322904Z       Hosting environment: Production
2023-08-14T23:43:20.7327575Z info: Microsoft.Hosting.Lifetime[0]
2023-08-14T23:43:20.7327822Z       Content root path: /home/vsts/work/1/s
2023-08-14T23:43:20.7338832Z dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
2023-08-14T23:43:20.7339439Z       Hosting started
2023-08-14T23:43:20.8201185Z info: Publisher[0]
2023-08-14T23:43:20.8201923Z       Processing modified files in commit ID...
2023-08-14T23:43:20.8960155Z info: Publisher[0]
2023-08-14T23:43:20.8962475Z       Putting logger apim-eventhub-logger...
2023-08-14T23:43:20.9041142Z dbug: PutRestResource[0]
2023-08-14T23:43:20.9042439Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/gi-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/gi-int-dv-apim/loggers/apim-eventhub-logger?api-version=2022-04-01-preview...
2023-08-14T23:43:31.5581855Z crit: Publisher[0]
2023-08-14T23:43:31.5583874Z       System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/gi-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/gi-int-dv-apim/loggers/apim-eventhub-logger?api-version=2022-04-01-preview failed with status code 500. Content is '{"error":{"code":"InternalServerError","message":"Request processing failed due to internal error.","details":null}}'.
2023-08-14T23:43:31.5584745Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
2023-08-14T23:43:31.5585496Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
2023-08-14T23:43:31.5585905Z          at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
2023-08-14T23:43:31.5586369Z       --- End of stack trace from previous location ---
2023-08-14T23:43:31.5586764Z          at publisher.Logger.PutLogger(LoggerName loggerName, JsonObject json, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-08-14T23:43:31.5587131Z          at publisher.Logger.<>c__DisplayClass8_0.<<ProcessArtifactsToPut>b__0>d.MoveNext()
2023-08-14T23:43:31.5587453Z       --- End of stack trace from previous location ---
2023-08-14T23:43:31.5587712Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2023-08-14T23:43:31.5588011Z       --- End of stack trace from previous location ---
2023-08-14T23:43:31.5588297Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
2023-08-14T23:43:31.5588747Z          at publisher.Logger.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-08-14T23:43:31.5590296Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-08-14T23:43:31.5590832Z          at publisher.Publisher.ProcessCommitIdFilesToPut(IReadOnlyCollection`1 commitIdFilesToPut, CancellationToken cancellationToken)
2023-08-14T23:43:31.5591175Z          at publisher.Publisher.RunWithCommitId(CommitId commitId, CancellationToken cancellationToken)
2023-08-14T23:43:31.5591542Z          at publisher.Publisher.Run(CancellationToken cancellationToken)
2023-08-14T23:43:31.5591796Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
2023-08-14T23:43:31.5624080Z info: Microsoft.Hosting.Lifetime[0]
2023-08-14T23:43:31.5624537Z       Application is shutting down...
2023-08-14T23:43:31.5627631Z fail: Microsoft.Extensions.Hosting.Internal.Host[9]
2023-08-14T23:43:31.5628412Z       BackgroundService failed
2023-08-14T23:43:31.5630345Z       System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/gi-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/gi-int-dv-apim/loggers/apim-eventhub-logger?api-version=2022-04-01-preview failed with status code 500. Content is '{"error":{"code":"InternalServerError","message":"Request processing failed due to internal error.","details":null}}'.
2023-08-14T23:43:31.5630985Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
2023-08-14T23:43:31.5631303Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
2023-08-14T23:43:31.5631866Z          at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
2023-08-14T23:43:31.5632220Z       --- End of stack trace from previous location ---
2023-08-14T23:43:31.5632556Z          at publisher.Logger.PutLogger(LoggerName loggerName, JsonObject json, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-08-14T23:43:31.5632910Z          at publisher.Logger.<>c__DisplayClass8_0.<<ProcessArtifactsToPut>b__0>d.MoveNext()
2023-08-14T23:43:31.5633227Z       --- End of stack trace from previous location ---
2023-08-14T23:43:31.5633531Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2023-08-14T23:43:31.5633849Z       --- End of stack trace from previous location ---
2023-08-14T23:43:31.5634135Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
2023-08-14T23:43:31.5634588Z          at publisher.Logger.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-08-14T23:43:31.5635194Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-08-14T23:43:31.5635687Z          at publisher.Publisher.ProcessCommitIdFilesToPut(IReadOnlyCollection`1 commitIdFilesToPut, CancellationToken cancellationToken)
2023-08-14T23:43:31.5636009Z          at publisher.Publisher.RunWithCommitId(CommitId commitId, CancellationToken cancellationToken)
2023-08-14T23:43:31.5636278Z          at publisher.Publisher.Run(CancellationToken cancellationToken)
2023-08-14T23:43:31.5636539Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
2023-08-14T23:43:31.5636973Z          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
2023-08-14T23:43:31.5637240Z dbug: Microsoft.Extensions.Hosting.Internal.Host[3]
2023-08-14T23:43:31.5637430Z       Hosting stopping
2023-08-14T23:43:31.5652943Z crit: Microsoft.Extensions.Hosting.Internal.Host[10

Also for your testing, did you create the logger as mentioned here https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials

waelkdouh commented 1 year ago

@Mohid-A if you resolved your issue please close the issue. As @guythetechie demonstrated he is not facing the issue when he tried to replicate it.

Mohid-A commented 1 year ago

Hi @waelkdouh I am still facing the issue and it's not resolved yet. I checked with @guythetechie if he followed the below steps provided by Microsoft and waiting for this response.

https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials

waelkdouh commented 1 year ago

We will get back to you when we get a chance to take a look. Please note that this OS an open source project. So we can't promise any timelines.

Patrick-Chojnacki-Snow commented 8 months ago

Hello @waelkdouh Has their been any progress with this issue as we are experiencing it aswell? Thank you!

waelkdouh commented 8 months ago

Can you please post the logs? Make sure you scrub them please for your privacy.

waelkdouh commented 8 months ago

@Mohid-A did you ever resolve this issue? @guythetechie cousint reproduce the issue.

Patrick-Chojnacki-Snow commented 8 months ago

2024-02-22T17:32:16.7591736Z info: Extractor[0] 2024-02-22T17:32:16.7592773Z Writing logger information file /tmp/apiops-extractor-sandbox/loggers/apim-logger-csdoopsaoai/loggerInformation.json... 2024-02-22T17:32:16.7593950Z info: Extractor[0]

image

The extractor tools extract all my APIM information including the loggers, but for the logger types EventHub it is missing these properties.

"endpointAddress": "didier-events.servicebus.windows.net/didier-hub", "identityClientId": "{{65d7af00000000000032}}",

Mohid-A commented 8 months ago

@Mohid-A did you ever resolve this issue? @guythetechie cousint reproduce the issue.

@waelkdouh I still do not have success on this. I have not got any response back on the below question

I checked with @guythetechie if he followed the below steps provided by Microsoft and waiting for this response.

https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials

waelkdouh commented 8 months ago

I will check with him again. But looking at the history of this conversation it seems like he was successful with his test.

guythetechie commented 8 months ago

@Mohid-A - I just tried again successfully with these settings.

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "adding a new logger with system assigned managed identity",
    "credentials": {
         "endpointAddress":"apimtsteh.servicebus.windows.net",
         "identityClientId":"SystemAssigned",
         "name":"hub1"
    }
  }
}

One other thing to check: have you given your APIM instance permissions on the Event Hub? It needs the Azure Event Hubs Data Sender role. image

Patrick-Chojnacki-Snow commented 8 months ago

We confirmed that when we send the REST API call it retrieves the correct properties, so that means APIM and EventHub are able to communicate properly and have the correct roles. The issue is on the tool itself, when writing the properties on the loggerInformation.json file.

Patrick-Chojnacki-Snow commented 7 months ago

Hi Everyone, Just wondering if we made any progress with this issue and what next steps might be looking like. Thank you

guythetechie commented 7 months ago

@Patrick-Chojnacki-Snow - the extractor will not extract the connection string and other sensitive information. The APIM REST API doesn't expose it via a GET request. You will have to pass that information by overriding your publisher configuration file. Something like this:

apis
- name: apiName
  properties:
    credentials:
      endpointAddress: youraddress
      connectionString: yourconnectionstring
Patrick-Chojnacki-Snow commented 7 months ago

We tried with that config but it did not work

joshuaauger commented 7 months ago

@guythetechie Following up on what @Patrick-Chojnacki-Snow said, we merged the suggestions into loggers/<logger>/loggerInformation.json as we aren't using the configuration file. We then saw this issue:

System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/loggers/<logger>?api-version=2022-04-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Validation failed for logger-id '<logger>'. Exception Received 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'","details":null}}'.