search

Yamato-Security / hayabusa

Hayabusa (隼) is a sigma-based threat hunting and fast forensics timeline generator for Windows event logs.
GNU Affero General Public License v3.0
2.26k stars 200 forks source link

Enhancement: add ETA to progress bar #1213

Closed YamatoSecurity closed 10 months ago

YamatoSecurity commented 11 months ago

I removed the ETA from the progress bar because it was highly inaccurate and fluctuate widely decreasing and increasing due to hayabusa calculating based on the number of evtx files it is currently processing instead of the number of total records. This is caused because sometimes file size can be very small for a file or very large. I would still like to keep the progress bar as is where we know the file name of the currently scanned evtx file but want to investigate if it is possible to quickly count the total number of records that need to be analyzed and then calculate the ETA based off that. This would also make the progress bar more accurate as it would update on every record scanned instead of based on files it finished scanning.

hitenkoku commented 11 months ago

To create this process, all records must first be read and counted, so the reading process will take a long time. To avoid consuming memory, we need a process that reads and counts only records, but the record reading time depends on hayabusa-evtx, so we will not know how long it will take until we try it. I will try to implement a prototype.

YamatoSecurity commented 11 months ago

@hitenkoku Thank you! I think the number of records are written in the evtx file headers so we just need to read those numbers from the different evtx files, not individually count all records. I am not sure how to call it from the evtx library though.

YamatoSecurity commented 10 months ago

Humm.. I looked into this and it seems that it is necessary to iterate through all the records in order to count them. I am going to close this issue for now..