Closed fukusuket closed 2 months ago
% ./hayabusa-2.14.0-mac-aarch64 csv-timeline -d ../all-evtx -o big.csv -w -s --debug
┏┓ ┏┳━━━┳┓ ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
by Yamato Security
Start time: 2024/06/16 14:32
Total event log files: 2,239
Total file size: 8.8 GB
Loading detection rules. Please wait.
Excluded rules: 20
Noisy rules: 12 (Disabled)
Deprecated rules: 204 (4.98%) (Disabled)
Experimental rules: 1,019 (24.88%)
Stable rules: 240 (5.86%)
Test rules: 2,836 (69.26%)
Unsupported rules: 45 (1.10%) (Disabled)
Hayabusa rules: 162
Sigma rules: 3,933
Total enabled detection rules: 4,095
Output profile: standard
Scanning in progress. Please wait.
[00:08:04] 2,239 / 2,239 [========================================] 100%
Scanning finished. Please wait while the results are being saved.
Rule Authors:
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (80) Nasreddine Bencherchali (61) frack113 (52) Florian Roth (35) │
│ oscd.community (28) Tim Shelton (14) Roberto Rodriguez @Cyb3r... (10) Daniil Yugoslavskiy (9) │
│ Roberto Rodriguez (9) OTR (8) Victor Sergeev (7) Timur Zinniatullin (7) │
│ Gleb Sukhodolskiy (5) Bhabesh Raj (4) Sander Wiebing (3) Wietze Beukema (3) │
│ Markus Neis (3) Jonhnathan Ribeiro (3) Michael Haag (3) Patrick Bareiss (2) │
│ Christopher Peacock @sec... (2) Ján Trenčanský (2) SOC Prime (2) Thomas Patzke (2) │
│ Sreeman (2) Oddvar Moe (2) James Pemberton@4A616D65... (2) KarneadesMarkus Neis (2) │
│ Anton Kutepov (2) Mark Woan (2) Center for Threat Inform... (2) Endgame (2) │
│ Jakob Weinzettl (2) Teymur Kheirkhabarov (2) @gott_cyber (2) JHasenbusch (2) │
│ Fukusuke Takahashi (2) Alexandr Yampolskyi (2) Swachchhanda Shrawan Poudel (2) SCYTHE @scythe_io (2) │
│ Samir Bousseaden (1) Matthew Green @mgreen27 (1) Luc Génaux (1) Ecco (1) │
│ Andreas Hunkeler (1) D3F7A5105 (1) Connor Martin (1) Stephen Lincoln @slincol... (1) │
│ Harish Segar (1) Eric Conrad (1) xorxes (1) Zach Stanford @svch0st (1) │
│ pH-T (1) Thurein Oo (1) Tim Rauch (1) Dimitrios Slamaris (1) │
│ FPT.EagleEye (1) @neu5ron (1) Open Threat Research (1) Cybex (1) │
│ Tom Kern (1) Aleksey Potapov (1) AlertIQ (1) X__Junior (1) │
│ Elastic (1) Anish Bogati (1) Cédric Hien (1) James Pemberton @4A616D6573 (1) │
│ Mark Russinovich (1) @redcanary (1) Timur Zinniatullin oscd.... (1) Yusuke Matsui (1) │
│ Joshua Wright (1) Maxime Thiebaut (1) xknow (1) @0xrawsec (1) │
│ Dmitry Uchakin (1) Natalia Shornikova (1) Max Altgelt (1) mdecrevoisier (1) │
╰─────────────────────────────────╌──────────────────────────────╌──────────────────────────────────╌─────────────────────────────────╯
Results Summary:
Events with hits / Total events: 2,451,184 / 6,611,184 (Data reduction: 4,160,000 events (62.92%))
Total | Unique detections: 2,504,040 | 269
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 18,160 (0.73%) | 34 (21.19%)
Total | Unique medium detections: 33,724 (1.35%) | 109 (25.65%)
Total | Unique low detections: 1,701,040 (67.93%) | 69 (40.52%)
Total | Unique informational detections: 751,116 (30.00%) | 57 (12.64%)
Dates with most total detections:
critical: n/a, high: 2023-11-06 (5,517), medium: 2023-11-06 (18,239), low: 2022-09-18 (912,894), informational: 2022-03-02 (206,023)
Top 5 computers with most unique detections:
critical: n/a
high: WinDev2310Eval (21), DESKTOP-A8CALR3 (8), DESKTOP-6D0DBMB (8), evtx-PC (7), Agamemnon (7)
medium: WinDev2310Eval (79), Agamemnon (36), DESKTOP-A8CALR3 (22), DESKTOP-6D0DBMB (21), evtx-PC (14)
low: WinDev2310Eval (43), DESKTOP-6D0DBMB (36), DESKTOP-A8CALR3 (30), Agamemnon (30), evtx-PC (19)
informational: WinDev2310Eval (41), DESKTOP-6D0DBMB (39), DESKTOP-A8CALR3 (37), WIN-TKC15D7KHUR (35), WIN-FPV0DSIC9O6.sigma.fr (34)
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts: Top high alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a File Creation Date Changed to Another Year (15,884) │
│ n/a Windows Shell/Scripting Application File Write to Sus... (991) │
│ n/a EVTX Created In Uncommon Location (986) │
│ n/a Proc Exec (Non-Exe Filetype) (60) │
│ n/a File Download with Headless Browser (60) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts: Top low alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (12,311) Proc Access (1,613,391) │
│ Process Ran With High Privilege (7,673) Possible Timestomping (71,065) │
│ Potential Credential Dumping Activity Via LSASS (6,135) Scheduled Task Created - Registry (8,185) │
│ LSASS Access From Program In Potentially Suspicious F... (2,396) Shell Context Menu Command Tampering (4,283) │
│ Uncommon New Firewall Rule Added In Windows Firewall ... (918) System Drawing DLL Load (1,038) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (542,438) Net Conn (14,234) │
│ File Deleted (94,703) Pipe Created (10,602) │
│ Pipe Conn (39,460) DNS Query (7,251) │
│ Proc Exec (23,695) WMI Provider Started (706) │
│ Proc Terminated (14,674) Suspicious High IntegrityLevel Conhost Legacy Option (322) │
╰──────────────────────────────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯
Saved file: big.csv (2.3 GB)
Elapsed time: 00:08:05.1846
Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240616_144044.log for details.
Rule Parse Processing Time: 00:00:01.752
Analysis Processing Time: 00:08:04.970
Output Processing Time: 00:00:00.099
Memory usage stats:
heap stats: peak total freed current unit count
reserved: 2.0 GiB 2.0 GiB 0 2.0 GiB
committed: 1.0 GiB 2.0 GiB 757.7 GiB -755.7 GiB ok
reset: 0
purged: 48.9 GiB
touched: 128.5 KiB 20.0 MiB 99.3 GiB -99.3 GiB ok
segments: 19 320 308 12 not all freed!
-abandoned: 1 1 0 1 not all freed!
-cached: 0 0 0 0 ok
pages: 0 0 1.1 Mi -1.1 Mi ok
-abandoned: 5 5 0 5 not all freed!
-extended: 0
-noretire: 0
mmaps: 0
commits: 0
resets: 0
purges: 21.0 Ki
threads: 17 17 1 16 not all freed!
searches: 0.0 avg
numa nodes: 1
elapsed: 486.869 s
process: user: 3184.333 s, system: 34.669 s, faults: 305, rss: 1.1 GiB, commit: 1.0 GiB
Elapsed time: 00:08:19.1011
Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240616_145404.log for details.
Rule Parse Processing Time: 00:00:01.621
Analysis Processing Time: 00:08:18.267
Output Processing Time: 00:00:00.099
Memory usage stats:
heap stats: peak total freed current unit count
reserved: 2.0 GiB 2.0 GiB 0 2.0 GiB
committed: 1.0 GiB 2.0 GiB 735.5 GiB -733.5 GiB ok
reset: 0
purged: 48.1 GiB
touched: 128.5 KiB 19.1 MiB 99.4 GiB -99.3 GiB ok
segments: 20 306 294 12 not all freed!
-abandoned: 1 1 0 1 not all freed!
-cached: 0 0 0 0 ok
pages: 0 0 1.1 Mi -1.1 Mi ok
-abandoned: 2 2 0 2 not all freed!
-extended: 0
-noretire: 0
mmaps: 0
commits: 0
resets: 0
purges: 23.6 Ki
threads: 17 17 1 16 not all freed!
searches: 0.0 avg
numa nodes: 1
elapsed: 499.984 s
process: user: 3262.851 s, system: 35.713 s, faults: 292, rss: 1.1 GiB, commit: 1.0 GiB
% ./hayabusa-2.16.0-mac-aarch64 csv-timeline -d ../all-evtx -o big.csv -w -s --debug
┏┓ ┏┳━━━┳┓ ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
by Yamato Security
Start time: 2024/06/16 15:03
Total event log files: 2,239
Total file size: 8.8 GB
Loading detection rules. Please wait.
Excluded rules: 20
Noisy rules: 12 (Disabled)
Deprecated rules: 204 (4.98%) (Disabled)
Experimental rules: 1,019 (24.88%)
Stable rules: 240 (5.86%)
Test rules: 2,836 (69.26%)
Unsupported rules: 45 (1.10%) (Disabled)
Hayabusa rules: 162
Sigma rules: 3,933
Total detection rules: 4,095
Creating the channel filter. Please wait.
Evtx files loaded after channel filter: 145
Detection rules enabled after channel filter: 4,061
Output profile: standard
Scanning in progress. Please wait.
[00:07:45] 145 / 145 [========================================] 100%
Scanning finished. Please wait while the results are being saved.
Rule Authors:
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (76) Nasreddine Bencherchali (61) frack113 (52) Florian Roth (35) │
│ oscd.community (28) Tim Shelton (14) Roberto Rodriguez @Cyb3r... (10) Daniil Yugoslavskiy (9) │
│ Roberto Rodriguez (9) OTR (8) Victor Sergeev (7) Timur Zinniatullin (7) │
│ Gleb Sukhodolskiy (5) Bhabesh Raj (4) Sander Wiebing (3) Wietze Beukema (3) │
│ Markus Neis (3) Jonhnathan Ribeiro (3) Michael Haag (3) Patrick Bareiss (2) │
│ Christopher Peacock @sec... (2) Ján Trenčanský (2) SOC Prime (2) Thomas Patzke (2) │
│ Sreeman (2) Oddvar Moe (2) James Pemberton@4A616D65... (2) KarneadesMarkus Neis (2) │
│ Anton Kutepov (2) Mark Woan (2) Center for Threat Inform... (2) Endgame (2) │
│ Jakob Weinzettl (2) Teymur Kheirkhabarov (2) @gott_cyber (2) JHasenbusch (2) │
│ Fukusuke Takahashi (2) Alexandr Yampolskyi (2) Swachchhanda Shrawan Poudel (2) SCYTHE @scythe_io (2) │
│ Samir Bousseaden (1) Matthew Green @mgreen27 (1) Luc Génaux (1) Ecco (1) │
│ Andreas Hunkeler (1) D3F7A5105 (1) Connor Martin (1) Stephen Lincoln @slincol... (1) │
│ Harish Segar (1) Eric Conrad (1) xorxes (1) Zach Stanford @svch0st (1) │
│ pH-T (1) Thurein Oo (1) Tim Rauch (1) Dimitrios Slamaris (1) │
│ FPT.EagleEye (1) @neu5ron (1) Open Threat Research (1) Cybex (1) │
│ Tom Kern (1) Aleksey Potapov (1) AlertIQ (1) X__Junior (1) │
│ Elastic (1) Anish Bogati (1) Cédric Hien (1) James Pemberton @4A616D6573 (1) │
│ Mark Russinovich (1) @redcanary (1) Timur Zinniatullin oscd.... (1) Yusuke Matsui (1) │
│ Joshua Wright (1) Maxime Thiebaut (1) xknow (1) @0xrawsec (1) │
│ Dmitry Uchakin (1) Natalia Shornikova (1) Max Altgelt (1) mdecrevoisier (1) │
╰─────────────────────────────────╌──────────────────────────────╌──────────────────────────────────╌─────────────────────────────────╯
Results Summary:
Events with hits / Total events: 2,450,614 / 6,463,018 (Data reduction: 4,012,404 events (62.08%))
Total | Unique detections: 2,503,458 | 265
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 18,160 (0.73%) | 34 (20.75%)
Total | Unique medium detections: 33,389 (1.33%) | 107 (26.04%)
Total | Unique low detections: 1,701,040 (67.95%) | 69 (40.38%)
Total | Unique informational detections: 750,869 (29.99%) | 55 (12.83%)
Dates with most total detections:
critical: n/a, high: 2023-11-06 (5,517), medium: 2023-11-06 (18,239), low: 2022-09-18 (912,894), informational: 2022-03-02 (206,005)
Top 5 computers with most unique detections:
critical: n/a
high: WinDev2310Eval (21), DESKTOP-A8CALR3 (8), DESKTOP-6D0DBMB (8), evtx-PC (7), Agamemnon (7)
medium: WinDev2310Eval (79), Agamemnon (35), DESKTOP-A8CALR3 (20), DESKTOP-6D0DBMB (20), evtx-PC (14)
low: WinDev2310Eval (43), DESKTOP-6D0DBMB (36), DESKTOP-A8CALR3 (30), Agamemnon (30), evtx-PC (19)
informational: WinDev2310Eval (39), DESKTOP-6D0DBMB (38), DESKTOP-A8CALR3 (37), WIN-TKC15D7KHUR (35), WIN-FPV0DSIC9O6.sigma.fr (32)
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts: Top high alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a File Creation Date Changed to Another Year (15,884) │
│ n/a Windows Shell/Scripting Application File Write to Sus... (991) │
│ n/a EVTX Created In Uncommon Location (986) │
│ n/a Proc Exec (Non-Exe Filetype) (60) │
│ n/a File Download with Headless Browser (60) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts: Top low alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (12,311) Proc Access (1,613,391) │
│ Process Ran With High Privilege (7,673) Possible Timestomping (71,065) │
│ Potential Credential Dumping Activity Via LSASS (6,135) Scheduled Task Created - Registry (8,185) │
│ LSASS Access From Program In Potentially Suspicious F... (2,396) Shell Context Menu Command Tampering (4,283) │
│ Uncommon New Firewall Rule Added In Windows Firewall ... (918) System Drawing DLL Load (1,038) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (542,438) Net Conn (14,234) │
│ File Deleted (94,703) Pipe Created (10,602) │
│ Pipe Conn (39,460) DNS Query (7,251) │
│ Proc Exec (23,695) WMI Provider Started (706) │
│ Proc Terminated (14,674) Suspicious High IntegrityLevel Conhost Legacy Option (322) │
╰──────────────────────────────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯
Saved file: big.csv (2.3 GB)
Elapsed time: 00:07:48.1457
Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240616_151108.log for details.
Rule Parse Processing Time: 00:00:01.587
Analysis Processing Time: 00:07:47.737
Output Processing Time: 00:00:00.105
Memory usage stats:
heap stats: peak total freed current unit count
reserved: 3.1 GiB 3.1 GiB 108.0 MiB 3.0 GiB
committed: 1.0 GiB 3.1 GiB 814.8 GiB -811.7 GiB ok
reset: 0
purged: 48.4 GiB
touched: 128.5 KiB 28.5 MiB 92.2 GiB -92.2 GiB ok
segments: 20 457 445 12 not all freed
-abandoned: 1 1 1 0 ok
-cached: 0 0 0 0 ok
pages: 0 0 1.1 Mi -1.1 Mi ok
-abandoned: 2 2 2 0 ok
-extended: 0
-noretire: 0
arenas: 3
-crossover: 0
-rollback: 0
mmaps: 0
commits: 0
resets: 0
purges: 33.7 Ki
threads: 17 17 1 16 not all freed
searches: 0.0 avg
numa nodes: 1
elapsed: 469.481 s
process: user: 3171.723 s, system: 34.336 s, faults: 316, rss: 1.1 GiB, commit: 1.0 GiB
Memory usage increased by 1GB with version 2.16.0. Thus, it appears to be an effect of the changes in version 2.16.0🤔
At the time the following PRs were merged, memory usage had already increased.
memo: memory usage:
@fukusuket Are you making sure you are testing with the same rules? I just tried with the current rules but version 2.14.0 does not support windash
so many parsing errors which results in less rules being loaded which is probably why less memory is being used. You might need to convert windash
rules to something compatible for 2.14.0 like we did in the past in order to do a proper comparison.
Also, when I am testing on my intel mac, the total committed amount of memory will change about 1GB so running 2.15.0 and 2.16.0 several times will sometimes result in 2GB and sometimes 3GB for each version. Sometimes 2.15.0 uses less memory but sometimes it will use 3GB and 2.16.0 uses only 2GB of memory. So I do not think this is reliable.
In my experiences, comparing the rss
and commit
values in the last line: process: user: 3171.723 s, system: 34.336 s, faults: 316, rss: 1.1 GiB, commit: 1.0 GiB
is usually more reliable for me.
@YamatoSecurity
Are you making sure you are testing with the same rules?
Yes! I am comparing using version 2.14.0 rules, so there is no difference in the number of rules when comparing. In my environment, the peak/reserved value is stable and I have been able to compare with this value without problems so far, but I will check!
@fukusuket I see! In that case, no problem. Thanks for looking into this!
Memory usage has increased by 1GB since the commit https://github.com/Yamato-Security/hayabusa/commit/94e8e19f5777e32a4e99ca0cd7e0b4e00fc09423 It appears that our code change is not the cause, but a version change in one of the dependent libraries...
There are so many changes in the dependent libraries that it is difficult to identify which library is responsible...😇
@fukusuket That's too bad. In my environment I do not notice a difference in memory usage so it may depend on the CPU architecture. If there is a regression, it might be nice to report it to the crate owner but I would rather stick to using the latest crate versions than use old crates unless there is really a significant degradation in performance or quality.
@YamatoSecurity Yes, I agree! Since it was not an implementation issue here, I will close it as an issue for now :)
Investigate the possibility of increased memory usage in later releases of the low-memory feature.