TraceLog: Allow custom user parsers

[dorian-apanel-intel]

It would be nice to add custom parsers to TraceLog. There is already a TODO for it: https://github.com/microsoft/perfview/blob/084c4de35943d511c844f40bcbdcaa0dd2c709c3/src/TraceEvent/TraceLog.cs#L912

[brianrob]

@dorian-apanel-intel, can you share some details about what you're hoping to do with this functionality?

[dorian-apanel-intel]

I have custom hand-written manifest and use it for logging in a native library and haven't found any popular C/C++ lib to parse ETL files.

After some adjustments, both to manifest and TraceParserGen, I was able to generate CustomTraceEventParser.

There are nice features of TraceLog/etlx mentioned in manual (going back/forth, iterating over one type of event, callstacks), but I cannot use it AFAIK without modifying TraceLog.cs)

PS: I also found this question: https://github.com/microsoft/perfview/issues/970 but second post does not look promising :)

[brianrob]

Thanks @dorian-apanel-intel. If I understand correctly, you're trying to parse your own events that don't come from an EventSource, where the manifest is written into the event stream. If I have that right, you should not need to add the parser when the TraceLog is produced. All events, regardless of whether or not TraceEvent knows how to consume them, will be copied. Then, when you access the event stream, you can use a parser that knows how to parse the events to consume them. You can create this parser using TraceParserGen, or you can register the manifest on the machine, and then use the RegisteredTraceEventParser, which will pick this up.

Do let me know if I've misunderstood.

[dorian-apanel-intel]

Sorry, I should be more verbose in previous post. Once I generated my parser using TraceParserGen, I have no problems with accessing my events in event stream (etl grabbed by xperf). Events are strong typed, so changing schema is less error prone - good.

What I would like to do next, is to try Higher Level Processing using TraceLog class. From reading this manual, I guess I cannot do:

using (var traceLog = TraceLog.OpenOrConvert("MyFile.etl")) {
    // For a particular process
    var p = traceLog.Processes.FirstProcessWithName("MyProc");
    foreach (var e in p.EventsInProcess.ByEventType<MyCustomEventTraceData>()) { ... }

because there is no way to pass my generated parser to TraceLog. Am I right?

[brianrob]

Yes, you can consume events from a TraceLog with any parser, including a custom one. To do this, you just need to get the underlying TraceEventSource from the TraceLog.

Something like this:

var source = traceLog.Events.GetSource();
var parser = new MyCustomParser(source);

parser.MyCustomEvent += delegate(MyCustomEventTraceData data)
{
  // Handle your event here.
};

I've not personally tried the code snippet from above, but if necessary, I can investigate and see if it would work.

// Process the trace, calling parser.MyCustomEvent each time an event of this type is encountered.
source.Process();

[dorian-apanel-intel]

Your snippet works, but let me quote manual to explain what I'm trying to do:

... Solving these problems is what the Microsoft.Diagnostics.Tracing.TraceLog class is all about. Unlike ETWTraceEventSource (which only supports the 'push' (callback) model) and supports real time streams. TraceLog ONLY deals with files, and can provide both a push (callback) model as well as a 'pull' (foreach) model. It also contains a rich object model that allows you to...

I would like to use this pull model.

Processing by callbacks works fine (push model). I don't know how to iterate back/forth through events by type/process (using TraceLog, without creating arrays on the side - pull model).

// pull model attempts are marked as "Never fired" in below snippet:

using (var traceLog = TraceLog.OpenOrConvert(etlFilePath)) // Cannot pass my parser to constructor+parse method
    {
        foreach (TraceEvent data in traceLog.Events)
        {
            Debug.WriteLine("Got Event {0}", data); // I see my events are raw here
        }

        // I would like this to work:
        foreach (var e in traceLog.Events.ByEventType<MyCustomTraceData>())
        {
            Debug.WriteLine("Got my event by iterating, first attempt");  // Never fired
        }

        // Create parser with source extracted from TraceLog:
        var source = traceLog.Events.GetSource();
        MyCustomTraceEventParser parser = new MyCustomTraceEventParser(source);

        // Lower level parsing by callbacks on stream
        parser.myCustomEvent += delegate (MyCustomTraceData data)
        {
            Debug.WriteLine($"Got my event by callbacks:{data.myCustomField}"); // Fired
        };

        source.Process(); // Invoke callbacks for events in the source

        // Higher level parsing by iterating of filtered events by type:
        foreach (var e in traceLog.Events.ByEventType<MyCustomTraceData>())
        {
            Debug.WriteLine("Got my event by iterating, second attempt");  // Never fired
        }
    }

[dorian-apanel-intel]

I have a fix that seems to work. It allows me to inject my parser and iterate over my events. (Custom action passed to TraceLog constructor) I will create PR one other with whitespaces in merged.

[dorian-apanel-intel]

For my personal use, this https://github.com/dorian-apanel-intel/perfview/commit/a0a9679a937f08757d5d627d9a19106fdc897a00 helps, but using TraceLog class and iterating back/forth is much less performant then creating own slimmed event arrays and iterating them. Therefore I've stopped using it for my use case.

@brianrob Do you want to merge it or shall we just close this issue?