[Windows] Improved support for Windows Event Collector

[jamiehynds]

Our Windows integration currently supports the 'Forwarded Events' channel, whereby events from numerous Windows hosts can reside. As outlined in this guide created by @ThorbenJ, there are several pitfalls to this approach.

Using Winlogbeat and a PowerShell script, it is possible to map forwarded events to individual event channels on a Windows Event Collector server. e.g. All PowerShell events forwarded by other hosts, are mapped to a PowerShell channel on the WEC server.

While this guide is based on Winlogbeat, we should consider how agent can support this architecture. Agent will need to auto-detect all the WEC channels that have been configured and generate a config accordingly.

[ThorbenJ]

Hi all,

All Windows Event Forwarding (WEF) are managed by Windows Event Collector (WEC) Subscriptions. Each Subscription defines the Channel (Log file) that all logs sent to that subscription are to be stored in. If you want more details, please read the guide that Jamie mentioned: https://ela.st/tj-qsg-wec

An Agent Integration that is interested in collecting all WEF events on a WEC server should enumerate all the Subscriptions and then follow/read the Channels (log files) bound by those subscriptions.

"wecutil.exe es" Will enumerate/list all subscriptions on the WEC server. "wecutil.exe gs NAME" will get all that subscriptions parameters. So to get all "Forward" logs one could do this with PowerShell:

& wecutil.exe es | ForEach-Object { (& wecutil.exe gs $_ | ConvertFrom-String -Delimiter : | Where-Object { $_.P1 -eq 'LogFile' }).P2.Trim() } 

What's nice about this is that it would work for any WEC architecture, not just the one laid out in my guide. Another popular WEC architecture is that from Palantir: https://github.com/palantir/windows-event-forwarding

As an FYI: Going back to my guide/architecture, all scripts are here: https://github.com/ElasticSA/wec_pepped - wec_config.ps1 is used to describe the WEC config particularly in regards to the local AD architecture. gen_winlogbeat_config.ps1 reads wec_config.ps1 to then generate the entire winlogbeat.event_logs: entry.

Using my guide WEC setup, the PS1 "wecutil" script snippet above would output:

WecFwdLog-Domain-Clients/Application
WecFwdLog-Domain-Clients/Misc
WecFwdLog-Domain-Clients/Script
WecFwdLog-Domain-Clients/Security
WecFwdLog-Domain-Clients/Service
WecFwdLog-Domain-Clients/Sysmon
WecFwdLog-Domain-Clients/System
WecFwdLog-Domain-Controllers/Application
WecFwdLog-Domain-Controllers/Misc
...
WecFwdLog-Domain-Privileged/Service
WecFwdLog-Domain-Privileged/Sysmon
WecFwdLog-Domain-Privileged/System
WecFwdLog-Domain-Servers/Application
WecFwdLog-Domain-Servers/Misc
WecFwdLog-Domain-Servers/Script
WecFwdLog-Domain-Servers/Security
WecFwdLog-Domain-Servers/Service
WecFwdLog-Domain-Servers/Sysmon
WecFwdLog-Domain-Servers/System 

On a Palentir WEC setup it would output:

WEC-Powershell
WEC-WMI
WEC-EMET
WEC-Authentication
WEC-Services
WEC-Process-Execution
WEC-Code-Integrity
WEC2-Registry
WEC2-Object-Manipulation
...
WEC6-Device-Guard
WEC6-ADFS
WEC6-Sysmon
WEC6-Software-Restriction-Policies
WEC6-Microsoft-Office
WEC7-Active-Directory
WEC7-Terminal-Services
WEC7-Privilege-Use

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ThorbenJ]

This has stalled so I thought I'd add another 2 pence to it.

On a positive note, we now have a custom event channel integration! Thank you! The negative would be you can only enter one channel, so you have to add and configure the integration multiple times. A WEC server following our recommendation guidelines would likely have several dozen channels.

Two great improvements, would be: 1) Allow listing multiple channels in a single custom windows integration. 2) A a specific WEC integration that automatically harvest from all channels that have a WEC subscription attached to them. My first comment above gave a powershell script to enumerate all such channels use the wecutil.exe command.

(cc @jamiehynds)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[novaksam]

+1 for multiple channels. I have tuned my event log buffers to account for the ingestion rate of different logs, and altered the file size for different event logs to account for the amount of data I collect in them. If I were to shunt everything into Forwarded Events I'm pretty sure things would get cycled out before they were read unless I made the log huge.