microsoft / krabsetw

KrabsETW provides a modern C++ wrapper and a .NET wrapper around the low-level ETW trace consumption functions.
Other
589 stars 149 forks source link

What API souuld I use to consume ETW? #44

Closed ghost closed 6 years ago

ghost commented 6 years ago

What API souuld I use to consume ETW:

  1. System.Diagnostics.Tracing.EventSource (BCL)
  2. Microsoft.Diagnostics.Tracing.EventSource (NuGet)
  3. Microsoft.Diagnostics.Tracing.TraceEvent (NuGet)
  4. krabsetw (NuGet) ?

I read this at https://blogs.technet.microsoft.com/office365security/hidden-treasure-intrusion-detection-with-etw-part-2/:

TraceEvent is a library used by the PerfView tool and has the benefits of being a well-designed .NET API. Unfortunately, it doesn’t perform well for scenarios where we want to keep memory usage to a minimum. System.Diagnostics.Tracing has the advantage of being part of the .NET BCL but we’ve observed intermittent exceptions and unexpected behavior in the past. Additionally, it suffers from the same memory consumption issue that TraceEvent does. In response to these challenges, Office 365 Security chose to implement our own API with three primary goals: •Intuitive and flexible API •High performance – filtering events in the native layer •Available both in .NET and native C++ The result of this work is krabsetw, a library we’ve open-sourced under the MIT license. It contains both a native C++ API as well as a .NET API. This library is used in production today across Office 365 workloads on more than 100,000 machines. With filtering, we’re able to process more than more than 500 billion events per day, generating more than 7TB of data per day across the machines.

Is this still true?.. It is written one year ago (May 9, 2017), and other API's have updates since. I want to know what API I can depend on, if not now, at least in upcoming updates.

Another question: Do I have to use these NuGets too? Microsoft.Diagnostics.Tracing.TraceEvent.SupportFiles Microsoft.Diagnostics.Tracing.EventRegister

zacbrown commented 6 years ago

Hi,

What you use will depend on your use-case. Ultimately, there's not much difference between System.Diagnostics.Tracing.EventSource and Microsoft.Diagnostics.Tracing.EventSource from my understanding. Microsoft.Diagnostics.Tracing.EventSource is just the .NET core iteration from what I see.

krabsetw was written with a specific use-case in mind: high performance real-time consumption of ETW events. The other options you listed (1 through 3) weren't found to be suitable for that use case when we turned on 10+ providers.

All of these APIs ultimately rely on the TDH APIs in Win32 which are not expected to be deprecated any time soon. They've been available since at least Windows XP and will likely be available moving forward for some time.

As to your last question - I'm not sure I understand. Generally, the NuGet will specify the dependencies needed so if you need to use a type, you add that NuGet or it will be added for you if it's a dependency?

krabsetw will continue to receive updates to bugs and possibly feature improvements when they're requested or when needed for the teams at Microsoft using the library. That said, it's not under active development as it is considered largely "feature complete" at this point.

ghost commented 6 years ago

@zacbrown Thanks a lot. By thevway, how do you pronounce krabsetw :D ?

zacbrown commented 6 years ago

Hi,

You'd basically pronounce it "crabs e t w". The origin of the name was "what's something easy to search for on Bing/Google?" - We didn't want to name it some generic sounding thing :).

-Z

zacbrown commented 6 years ago

Closing this question as it appears addressed.