Windows Event Log Parser

[nigleweber]

Event Log Parser

Let's build a Windows Event Log Parser to help with assessing a system under an IR investigation. You will need several modules to make this project readable, maintainable, functional and most importantly extensible.

Part 1 - Parse the logs

To start with we need to sift through all the event logs gathering the relevant/important ones along with the time the event occurred to help build up a timeline of activity on the system. To do so we'll need to define a list of event logs that are of interest to us. The logs should be returned as list/array/dictionary, this is relevant for Part 4.

Part 2 - Analysis Engine

Once we have the event logs we'll want to have an analysis layer to interpret our findings as the number of logs is expected to be quite large. To keep the scope of this engine reasonable let's have it run with "rules" defined as arrays. These rules will be a set of tuples with two values: event name/id keys and a "maximum time window allowed until the next event in the chain occurs. Index 0 will contain the name of the rule. We can keep track of the rules that triggered in an array or dictionary, where we take the timestamp of the first event as the "time the rule triggered".

Part 3 - Output

This module will be responsible for formatting and outputting the results of the analysis engine as well as any errors or anomalies found in the parsing module.

Part 4 - Testing Module

Since your machine is unlikely to have many if any suspicious activities on it, we need to build a small test loader. It should have pre-defined event log arrays/lists/dictionaries to showcase the various rules we create. This will allow you to add and test new rules in a seamless fashion.

[nigleweber]

Event Log Parser step:

• [ ] Read up on event logs and build an initial list of events IDs that are of interest

• [ ] Create a main module <- this will be largely empty for most of the project

  • [ ] Parse/handle user input here <- should only happen at the start of the program
  • [ ] Take a user supplied start and end time to scan through

• [ ] Build a parser module that is called by the main module

• [ ] Make a json file with all the interesting event IDs and names <- helps with extensibility

  • [ ] each event should be on it's own line so we can add comments inline

• [ ] Make a second json file with the rules <- helps with extensibility

  • [ ] each rule should be on it's own line

• [ ] Parser Module

  • [ ] Load the interesting events list
  • [ ] Open and read all the event logs for the user supplies time range
  • [ ] For each event add it to a new list "found_interesting_events" or some such IF it matches the interesting list
  • [ ] Return the found interesting events list to main

• [ ] Analysis Engine

  • [ ] Load the rules
  • [ ] Iterate through each element in the list using a while/for loop
  • [ ] Iterate through all known rules, if the event matches the start of a rule begin processing that rule
    • [ ] Check if the next event in the rule is present within the defined time window if true - continue
    • [ ] If false - break

• [ ] Output module

  • [ ] Dump list of all interesting events to file
  • [ ] Write matching rule triggers to file

Note for analysis engine: The analysis phase will consist of at least 3 nested loops. I strongly suggest breaking each loop into it's own function. This is also a good idea in general. Python will convince you this is not necessary, however come golang re-write time you'll be glad you learned this. ;)

[nigleweber]

Analysis Engine - Pseudo code - IN PROGRESS

def analysis_engine(list_of_events_ordered_sequentially, list_of_rules):
detections = []
for index, event in list_of_events_ordered_sequentially:
  for rule in list_of_rules:
    if event.ID == rule.ID:
      triggered_rule = process_rule(list_of_events_ordered_sequentially, event, index, rule)
      if triggered_rule != "":
        detections.append(triggered_rule)
return detections

def process_rule(list_of_events_ordered_sequentially, event, index, rule):
detection = ""
if rule.type == "single_event":
  detection = {rule.name: (event.timestamp, event.human_readable_time)}
elif rule.type == "count":
  window_limited_list = get_events_within_time_window(list_of_events_ordered_sequentially, index, rule.window)
  detection = process_counting_rule(window_limited_list, rule)
elif rule.type == "ordered":
  window_limited_list = get_events_within_time_window(list_of_events_ordered_sequentially, index, rule.window)
  detection = process_ordered_rule(window_limited_list, rule)
elif rule.type == "unordered":
    window_limited_list = get_events_within_time_window(list_of_events_ordered_sequentially, index, rule.window)
  detection = process_unordered_rule(window_limited_list, rule)
return detection

def get_events_within_time_window(list_of_events_ordered_sequentially, index, rule):
start_event = list_of_events_ordered_sequentially[index]
end_time = start_event.timestamp + rule.window
start_index_list = list_of_events_ordered_sequentially[index+1:] # Start at the next index since we don't want to re-process the same event
for event in start_index_list:
  if event.timestamp > 

# This function should iterate through the 
def process_counting_rule(window_limited_list, rule):

return detection

[nigleweber]

Analysis Pseudo Code - Complete

This code mostly completes what's needed for the analysis engine. Obviously it needs testing/transformation into runnable code. Like I mentioned we need to expand the number of rules and events that we monitor. I recommend going through the list of all events and looking for ordered/unordered combos and then adding the required events.

@v3r1t4s Do let me know if you're feeling lost or stuck on this part as it's not at all trivial.

# The core of the analysis engine
def analysis_engine(list_of_events_ordered_sequentially, list_of_rules):
  detections = []
  for index, event in list_of_events_ordered_sequentially:
    for rule in list_of_rules:
      if event.ID == rule.ID:
        triggered_rule = process_rule(list_of_events_ordered_sequentially, event, index, rule)
        if triggered_rule != "":
          detections.append(triggered_rule)
  return detections

# Processes rules for the analysis engine
def process_rule(list_of_events_ordered_sequentially, event, index, rule):
  detection = ""
  if rule.type == "single_event": # Since we're in this function, we automatically know we matched the first item in a rule
    detection = {rule.name: (event.timestamp, event.human_readable_time)}
  elsif rule.type == "count": # Same event multiple times
    detection = process_counting_rule(list_of_events_ordered_sequentially, event, index, rule)
  elsif rule.type == "ordered": # A series of events, where all of them are in order
    detection = process_ordered_rule(list_of_events_ordered_sequentially, event, index, rule)
  elsif rule.type == "unordered": # A series of events where the first is "ordered" and the rest can happen in any order within the window
    detection = process_unordered_rule(list_of_events_ordered_sequentially, event, index, rule)
  return detection

# Helper function that sets up all rule processing functions
def rule_preprocessor(list_of_events_ordered_sequentially, event, index, rule):
  detection = ""
  end_time = event.timestamp + rule.window
  events_within_window = get_events_within_time_window(list_of_events_ordered_sequentially, event, index, rule)
  return detection, end_time, events_within_window

# Trims the list to only have the elements within the given timewindow
def get_events_within_time_window(list_of_events_ordered_sequentially, start_event, index, rule):
  end_time = start_event.timestamp + rule.window
   start_index_list = list_of_events_ordered_sequentially[index+1:] # Start at the next index since we don't want to re-process the same event
  for index, event in start_index_list:
    if event.timestamp > end_time:                                 # This event is outside the Window
      return start_index_list[:index-1]                            # Return the list excluding this event and any occurring after it
  return start_index_list                                          # Should only reach here if end_time exceeds the last event timestamp

# This function should iterate through the events starting with the first match
def process_counting_rule(list_of_events_ordered_sequentially, event, index, rule):
  detection, end_time, events_within_window = rule_preprocessor(list_of_events_ordered_sequentially, event, index, rule)
  count = 1 # Starts count at one since we already have the first match

  for window_event in events_within_window:
    if window_event.id == rule.id:
      count = count + 1
  if count >= rule.required_count:
    {rule.name: (event.timestamp, event.human_readable_time)} # We can think about adding each event into this list
  return detection

# This function iterates through the events and looks for an ordered sequence of events
def process_ordered_rule(list_of_events_ordered_sequentially, event, index, rule):
  detection, end_time, events_within_window = rule_preprocessor(list_of_events_ordered_sequentially, event, index, rule)
  rule_index = 1                         # We need the event ID for the second event in the list of events
  current_rule_id = rule.ids[rule_index] # Obtain the event ID for the next event we expect to see in the ordered list
  len_rule_ids = len(rule.ids)

  for window_event in get_events_within_time_window:
    if window_event.id == current_rule_id:
      rule_index = rule_index + 1
      if rule_index >= len_rule_ids:    # We matched the last event needed to trigger the rule
        detection = {rule.name: (event.timestamp, event.human_readable_time)} # We can think about adding each event into this list
        return detection
      else:                             # We can continue processing
        current_rule_id = rule.ids[rule_index] # Obtain the event ID for the next event we expect to see in the ordered list
  return detection

# This function iterates through the events and looks for a set of events
def process_unordered_rule(list_of_events_ordered_sequentially, event, index, rule):
  detection, end_time, events_within_window = rule_preprocessor(list_of_events_ordered_sequentially, event, index, rule)
  rule_ids = rule.ids           # Obtain a copy of the list of events the rule needs to see

  for window_event in get_events_within_time_window:
    if window_event.id in rule_ids:
     rule_ids.remove(window_event.id) # Keep removing matched items until there are none left
  if len(rule_ids) < 1:
    detection = {rule.name: (event.timestamp, event.human_readable_time)} # We can think about adding each event into this list
  return detection

##### Structure for rules
{
    "rules": [
        {
            "Type": Counting
            "ID": 1100,
            "Required_Count": 10,
            "action": "alert",
            "description": "The event logging service has shutdown. This could indicate a problem with the logging system or a potential security issue."
        },
        {
            "Type": Ordered
            "ID": 1105
            "IDs": [1105, 1110, 1020, 1562],
            "action": "alert",
            "description": "The audit log was cleared. This could indicate an attempt to cover up malicious activity."
        },
        {
          "Type": Unordered
          "ID": 1105
          "IDs": [1105, 1110, 1020, 1562],
          "action": "alert",
          "description": "The audit log was cleared. This could indicate an attempt to cover up malicious activity."
      }
      ]
    }