Feature Request: Azure Event Hub and Even Management

[archmangler]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Configuration and Configuration Framework to create a centralised Azure Eventhub for streaming events from infrastructure to an eventhub in the management subscription. Configuration scheme to allow event-subscriptions to be created as well as filtering of events into/from the eventhub.

Is your feature request related to a problem?

Need for a centralised event hub in Azure which can be used to integrate with external SIEM.

Describe the solution you'd like

• A default central event hub created in the management subscription.
• By default set to collect some basic Azure infrastructure events from landing zone subscriptions.
• Either set to configure events and collect all events from the Landing Zones deployed using the module - or -
• providing the configuration method to configure events to be sent to the event hub from new landing zones

Alternative currently available:

Manual creation of eventhub using an external terraform module.

Additional context

It seems reasonable that centralised event hub should be part of the landing zone definition for Azure CAF.

[krowlandson]

This is an interesting ask, thank you for raising this @archmangler.

We are currently working with another team internally to migrate our diagnostic policies to built-in. As part of this, that team are creating equivalent policies for sending logs to Log Analytics workspace and Event Hub.

Once these are available, we plan to update our assignments to use the built-in policies and will deprecate our custom policies. As part of this work, it would seem reasonable to consider an option for Event Hub so I will discuss this with the team and provide feedback.

[krowlandson]

@archmangler... just to follow up on this.

I have discussed this with the team and some concerns were raised around the scalability of this approach. We have had some customers who experienced throughput constraints with the event hub, making the idea of a single centralised event hub something which we cannot currently recommend. This is also why we recommend using Log Analytics workspace.

However we recognise the reality that not all customers will necessarily hit these constraints and event hubs are a necessary integration point for some 3rd party solutions to integrate with Azure logs.

For now we don't plan to add this as an option for Azure landing zones but we will take this as feedback and I will investigate this further with the PG. We have an on-going engagement with them regarding the new built-in policies I mentioned above which should help us get a more concrete answer.

cc: @paulgrimley to raise this with the PG as to what their view is on this given the upcoming built-in policies for sending logs to event hub, and the customer observed throughput issues.

[paulgrimley]

@archmangler, @krowlandson and myself have spoken with those rolling out built-in policies for diagnostic logging and they confirmed there are plans for Event Hubs also - these are planned to be available early next year all being well

[krowlandson]

@archmangler... we are continuing to discuss this internally to establish the best practices and determine viability of including as an option within our reference implementations.

Previously we have only recommended Log Analytics workspaces over event hub due to the additional capabilities this offers. Of course we recognise that customers also want to integrate with 3rd party solutions.

When doing this, it's important that you correctly monitor and scale your event hub to avoid data loss. The product group added the When to scale my dedicated cluster? section to their documentation to help customers with this type of scenario. Once the built-in policies are available, this should be relatively simple to setup.

Because setting up event hub requires right-sizing, I cannot say at this point whether we will include it in the reference implementations.

We will continue to keep this issue open for tracking whilst we make a decision on this.

Thank you again for raising this 👍🏻

[archmangler]

Thanks for the update @krowlandson. If LA workspace offers the same capabilities, particularly in terms of integration with third party solutions like Confluent Kafka and Cloudera Data Platform then I'm sure it can supersede eventhub.