search

Yamato-Security / hayabusa

Hayabusa (隼) is a sigma-based threat hunting and fast forensics timeline generator for Windows event logs.
GNU Affero General Public License v3.0
2.26k stars 200 forks source link

Bug: metrics, logon-summary and csv-timline show different total event records #1105

Closed YamatoSecurity closed 1 year ago

YamatoSecurity commented 1 year ago

When running the metrics, logon-summary and csv-timeline commands, the total event records count will be different. Test: ./hayabusa-2.6.0-mac-intel metrics -d ../hayabusa-sample-evtx -o test.csv -C

./hayabusa-2.6.0-mac-intel logon-summary -d ../hayabusa-sample-evtx -o test.csv -C

./hayabusa-2.6.0-mac-intel csv-timeline -d ../hayabusa-sample-evtx -o test.csv -C

Results: logon-summary: 94950 metrics: 47475 csv-timeline: 47,476

Also, there are no commas displayed with logon-summary and metrics so it is difficult to read the number when the event record count is in the millions. (ie. 234723049) So I would like to add commas like in csv-timeline (-> 234,723,049)

@hitenkoku Could you look at this whenever you have time? (I think you are the most knowledgable about this)

hitenkoku commented 1 year ago

` @YamatoSecurity

The cause of the problem has been identified. The logon-summary is double the number of records in metrics because it was double counted with the metrics tally.

The difference between metrics and csv-timeline is that the record count in metrics was counting the number of records after channel name and EID filtering. The number in csv-timeline is the original record count.

Filtered record is following, filtered by Event>System>Channel is not found.

{"Event": Object {"SubscriptionBookmarkEvent": Object {"SubscriptionId": Null}, "System": Object {"Computer": String("dhcp01.offsec.lan"), "EventID": String("111"), "Provider_attributes": Object {"Name": String("Microsoft-Windows-EventForwarder")}, "TimeCreated_attributes": Object {"SystemTime": String("2021-04-27T15:03:16.983Z")}}}, "Event_attributes": Object {"xmlns": String("http://schemas.microsoft.com/win/2004/08/events/event")}}
hitenkoku commented 1 year ago

@YamatoSecurity There are two ways to fix it: 1. for 2. it affects the % of metrics and logon-summary display; and 2. for 3. it affects the % of logon-summary display. If you want to avoid affecting the % notation, need to separate the display variables from the aggregate variables. (In #1106, This adjusted)

  1. count the number of records in csv-timeline as the number of records after filtering for Channel and EID, instead of the total number of records
  2. the number of records in logon-summary and metrics should be the same as the overall number of records used in csv-timeline (this is currently done in the implementation at https://github.com/Yamato-Security/hayabusa/pull/1106) currently implemented at )