search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
194 stars 420 forks source link

[Azure] [Microsoft Entra ID] Allow Separate Eventhub and Storage Container Configurations per Log Type #10462

Open lucian-ioan opened 2 months ago

lucian-ioan commented 2 months ago

Currently, we assume the customer will have one eventhub with multiple log types ingested when configuring the integration. If not, we expect them to configure the integration multiple times for each log type (and eventhub).

The goal of this issue is to provide an option from the UI to ingest from multiple eventhubs automatically, as well as provide further assistance in case one eventhub is a bottleneck.

This might not be straightforward, as an agent is tied to an integration and this will spawn multiple agents at once.

hullarb commented 1 month ago

Hi Lucian,

The main issue i see with the current implementation is that all four different log types will have a separate consumer reading the single eventhub independently and they will reuse the same consumer group that is against the recemondation (having separate consumer group per consumer) and might cause inconsistent results.