add source: Windows Events

[Hoverbear]

Source: https://gitter.im/timberio-vector/community?at=5ed5dd5e4c9b0f060d3b616a

This started with a user is reporting issues with the File source on Windows, specifically the Fingerprinter. Unfortunately these files seem to be compressed and non-utf-8. We may need to add a specific source for this. It seems to be a WIndows service we could access via Winapis?

They were trying to do this:

 [sources.in]
include = ["C:\Windows\Logs*"] # required
type = "file" # required

[sinks.out]
encoding.codec = "ndjson" # required
inputs = ["in"] # required
path = "vector-%Y-%m-%d.log" # required
type = "file" # required 

I then recommended:

 [sources.in]
include = ["C:\\Windows\\Logs*"] # required
type = "file" # required

[sinks.out]
encoding.codec = "ndjson" # required
inputs = ["in"] # required
path = "vector-%Y-%m-%d.log" # required
type = "file" # required 

This did not resolve the problem.

C:\Program Files\Vector>.\bin\vector.exe --config config\vector.toml
Jun 02 05:11:10.281 INFO vector: Log level "info" is enabled.
Jun 02 05:11:10.297 INFO vector: Loading configs. path=["config\vector.toml"]
Jun 02 05:11:10.297 INFO vector: Vector is starting. version="0.9.1" git_version="v0.9.1" released="Thu, 30 Apr 2020 15
:49:24 +0000" arch="x86_64"
Jun 02 05:11:10.312 INFO vector::topology: Running healthchecks.
Jun 02 05:11:10.312 INFO vector::topology: Starting source "in"
Jun 02 05:11:10.312 INFO vector::topology: Starting sink "out"
Jun 02 05:11:10.312 INFO vector::topology::builder: Healthcheck: Passed.
Jun 02 05:11:10.312 INFO source{name=in type=file}: vector::sources::file: Starting file server. include=["C:\Windows\
\Logs\"] exclude=[]
Jun 02 05:11:10.328 ERROR source{name=in type=file}:file_server: file_source::file_server: Error reading file for finger
printing err=Access is denied. (os error 5) file="C:\Windows\Logs\BPA" 

Some more verbose output:

Jun 02 05:20:42.980 TRACE sink{name=out type=file}: vector::sinks::util::sink: Acking events. acking_num=0
Jun 02 05:20:42.980 ERROR source{name=in type=file}:file_server: file_source::file_server: Error reading file for finger
printing err=Access is denied. (os error 5) file="C:\Windows\Logs\BPA"
Jun 02 05:20:42.980 ERROR source{name=in type=file}:file_server: file_source::file_server: Error reading file for finger
printing err=Access is denied. (os error 5) file="C:\Windows\Logs\CBS"
Jun 02 05:20:42.980 ERROR source{name=in type=file}:file_server: file_source::file_server: Error reading file for finger
printing err=Access is denied. (os error 5) file="C:\Windows\Logs\DISM"
Jun 02 05:20:42.980 ERROR source{name=in type=file}:file_server: file_source::file_server: Error reading file for finger
printing err=Access is denied. (os error 5) file="C:\Windows\Logs\DPX"
Jun 02 05:20:42.980 ERROR source{name=in type=file}:file_server: file_source::file_server: Error reading file for finger
printing err=Access is denied. (os error 5) file="C:\Windows\Logs\MeasuredBoot"
Jun 02 05:20:42.996 INFO source{name=in type=file}:file_server: file_source::file_server: Found file to watch. path="C:
\Windows\Logs\DirectX.log" file_position=664874

These service files are not plaintext or easily accessible:

[Ulfy]

I would love to see broader Windows support, especially for Event Logs.

[clay584]

I'd also love to see Windows event logging support. I think there is a huge community that would gladly migrate from beats/logstash to Vector if there were support for this. I currently run lots of collectors for an MSSP that are all built on the aforementioned platforms, but with Elastic's recent licensing change, we are looking to move. There are a lot of great benefits by going with Vector, but lack of Windows event logging support is keeping us from adopting.

BTW, I absolutely love the pipeline unit testing. Such a great feature! 💯

[ypid-geberit]

I see adding support for the Windows event log as a can of worms. You have no idea what you are getting into when you start doing that. Let me only refer you to https://github.com/elastic/beats/issues/16334. Winlogbeat is doing/planning to do a lot like GUID lookups and so on that are best done by the agent on the system that emits the logs. I plan to use Vector as collection pipeline for almost anything, including Windows, but not for the event log.

[ypid-geberit]

This issue seems to be a duplicate of #1206, I guess one of the two should be closed. Probably the one that was opened after the other, so this (#2719) one.

[jszwedko]

Agreed, this is a duplicate of #1206, thanks @ypid-geberit.