search

Yamato-Security / hayabusa

Hayabusa (éš¼) is a sigma-based threat hunting and fast forensics timeline generator for Windows event logs.
GNU Affero General Public License v3.0
2.32k stars 203 forks source link

[bug] High memory usage even in `low-memory mode` #1438

Closed fukusuket closed 1 month ago

fukusuket commented 1 month ago

Describe the bug I noticed that even in low-memory mode, since 2.16.0, memory usage is high ... :(
I'll investigate if I can reduce memory usage. 🤔

I compared them with the results of the baseline-evtx and 2.14.0's rule. Ver low-memory Memory usage(peak-reserved) Memo
2.14.0 Y 2.0 GiB low-memory mode was released
2.15.0 Y 3.0 GiB https://github.com/Yamato-Security/hayabusa/issues/1368 1GB increase is a crate side issue.
2.16.0-dev Y 5.2 GiB d578a8969c1fae6857efc0417d548935bcd31453 https://github.com/Yamato-Security/hayabusa/pull/1334
2.16.0-dev Y 5.2 GiB 48fa7e0bbb5b2ca5688354e89b5be13ada0513d4 https://github.com/Yamato-Security/hayabusa/pull/1336
2.16.0-dev Y 5.2 GiB 69b7793d1d9266d15d58c6381e147227642ca179 https://github.com/Yamato-Security/hayabusa/pull/1347
2.16.0-dev Y 5.2 GiB b8b0f86254e2c5146170dcb40647aa75f6563c00 https://github.com/Yamato-Security/hayabusa/pull/1349
2.16.0-dev Y 5.2 GiB 310756f788c73de4ecb6b2b02ab9cc720be0513a https://github.com/Yamato-Security/hayabusa/pull/1352
2.16.0 Y 5.2 GiB
2.17.0 Y 5.2 GiB low-memory mode enabled by default
2.18.0-dev Y 5.2 GiB
2.18.0-dev Y 5.2 GiB -A -a option

The results of the execution are attached. perf.txt

Expected behavior Memory usage to be about the same as 2.15.0

fukusuket commented 1 month ago

Released feature in 2.16.0

fukusuket commented 1 month ago

I checked out->build commits since 2.15.0 and confirmed that the following PR seems to be the cause.

The following measurement results show that the problem seems to be that memory usage is high only in low-memory mode.

YamatoSecurity commented 1 month ago

I confirmed there is a memory increase: I tested with 14GB sample files:

v2.14.0 with low memory mode:

Rule Parse Processing Time: 00:00:06.643
Analysis Processing Time: 00:37:27.972
Output Processing Time: 00:00:00.385

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     4.0 GiB     4.0 GiB     0           4.0 GiB
 committed:     1.6 GiB     4.0 GiB  2175.2 GiB -2171.2 GiB                          ok
     reset:     0
    purged:   151.0 GiB
   touched:   128.5 KiB    35.3 MiB  1546.9 GiB -1546.9 GiB                          ok
  segments:    31         565         553          12                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0           7.5 Mi     -7.5 Mi                           ok
-abandoned:     4           4           0           4                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    80.8 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:  2255.164 s
   process: user: 22488.307 s, system: 314.314 s, faults: 0, rss: 2.9 GiB, commit: 1.6 GiB

main branch:

Rule Parse Processing Time: 00:00:06.346
Analysis Processing Time: 00:28:35.124
Output Processing Time: 00:00:00.521

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     7.3 GiB     7.3 GiB   384.0 MiB     7.0 GiB
 committed:     1.5 GiB     7.3 GiB  2026.6 GiB -2019.3 GiB                          ok
     reset:     0
    purged:   113.9 GiB
   touched:   128.5 KiB    42.3 MiB  1220.4 GiB -1220.4 GiB                          ok
  segments:    31         678         665          13                                not all freed
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           7.1 Mi     -7.1 Mi                           ok
-abandoned:     3           3           3           0                                ok
 -extended:     0
 -noretire:     0
    arenas:     7
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    54.5 Ki
   threads:    33          33           1          32                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:  1722.236 s
   process: user: 17143.321 s, system: 222.957 s, faults: 0, rss: 2.9 GiB, commit: 1.5 GiB

main branch with sorting events:

Rule Parse Processing Time: 00:00:06.132
Analysis Processing Time: 00:25:09.594
Output Processing Time: 00:04:49.126

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    21.0 GiB    23.5 GiB     5.5 GiB    18.0 GiB
 committed:     1.6 GiB    23.5 GiB  1965.1 GiB -1941.6 GiB                          ok
     reset:     0
    purged:   101.0 GiB
   touched:   128.5 KiB   206.9 MiB  1122.2 GiB -1122.0 GiB                          ok
  segments:    22           3.2 Ki      3.2 Ki     11                                not all freed
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           6.6 Mi     -6.6 Mi                           ok
-abandoned:     3           3           3           0                                ok
 -extended:     0
 -noretire:     0
    arenas:    13
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    68.4 Ki
   threads:    33          33           1          32                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:  1808.210 s
   process: user: 18140.995 s, system: 251.570 s, faults: 0, rss: 19.2 GiB, commit: 1.6 GiB
YamatoSecurity commented 1 month ago

@fukusuket I tested things monitoring with Activity Monitor and found out that the maximum memory usage is not in the top lines of reserved and committed but in the bottom line rss. In v2.14.0 and the latest branch, my maximum memory usage was 2.9GB for the 14GB files so memory usage is actually the same. (When sorted, it goes up to 19GB)

So I don't think there is a problem. But if you want to check on your end, please be sure to also monitor with Activity Monitor or Task Manager.

fukusuket commented 1 month ago

@YamatoSecurity Thank you so much for checking! I see, I had been checking peak/reserved until now. This does not seem to represent the maximum memory usage now. I am relieved to hear that looks fine...

I 'll check with Windows Task Manager and if everything is OK, close this issue!

fukusuket commented 1 month ago

@YamatoSecurity I checked the low-memory mode in Windows task manager and compared 2.15.0 with the main branch and found that the low-memory mode worked fine!

As you said, the rss value was close to the maximum memory usage, and both had a maximum usage of 1.4 GB. No memory issue were found and the issue is closed! Sorry for the misunderstanding😅