Iterating over aaLog file records question

[kuraara]

Hi,

I'm experimenting with the aaLogReader class and to get familiar with it I'm doing some basic CSV conversion to understand it better. The code I've written seems to work relatively okay, however I'm finding that the number of records processed exceeds 2 million entries for a 10MB aaLOG file.

So my two questions are:

1. Am I using the class correctly (albeit slightly hacky)?
2. Is 1 million+ records normal, or have I encountered some sort of circular reference loop?

    var reader = new aaLogReader();
    reader.OpenLogFile(LogFilePath);
    var records = reader.GetUnreadRecords();
    int n = 0;
    while (reader != null) {

    records = reader.GetUnreadRecords();
    foreach (var r in records)
    {

        n++;
        string msg = r.Message;
        MatchCollection ms = reg.Matches(r.Message);
        string opc = "";
        string tag = "";

        /**** 
            <!-- Doing some stuff with the message string to make them more compact -->
        *****/

        // Cache file output and only write to output streams every 1000 records
        if (n % 1000 == 0)
        {
            Console.Write("Writing log item {0}\r", n);
            file.Write(chunk);
            chunk = String.Format("\n{0},{1},{2},{3},{4},{5}", r.HostFQDN, GetUnixTime(r.EventDateTimeUtc), r.LogFlag, r.ProcessID, r.ProcessName, msg.Replace("\n", ";").Replace("\r", ""));
        }
        else
        {
            chunk += String.Format("\n{0},{1},{2},{3},{4},{5}", r.HostFQDN, GetUnixTime(r.EventDateTimeUtc), r.LogFlag, r.ProcessID, r.ProcessName, msg.Replace("\n", ";"));
        }

    }
    }
    file.Write(chunk);

[arobinsongit]

Hey - so happy you are finding some use in the library.

At the moment I don't quite have time to take your code and run it as I'm deployed at a customer site but at first glance it does seem you have a couple nexted loops going on. The library was written to try to simplify the consumers logic on the outside.

I would go check this example https://github.com/aaOpenSource/aaLog/blob/master/aaLogSavetoFileExample/MainForm.cs

as it might be in the ballpark if what you are trying to do.

But the basic concept is Initialize a log reader

logReader = new aaLogReader.aaLogReader();

and then call GetUnreadRecords List<LogRecord> records = logReader.GetUnreadRecords();

Every time you call GetUnreadRecords it creates a cache file indicating what the last record read was. So next time you call it you only get the new records. There is a default number of records it returns (1000). If you want more than this then you can override this value.

If you want to go back to a specific time or a specific number of records regardless of read status there are numerous other forms of the Get records call that give you a lot of flexibility.

Hopefully this helps.

-Andy