feat: load config files from a single file

What Changed

Closed #1420
Related https://github.com/Yamato-Security/hayabusa-encoded-rules/issues/2

Specification

loading config logic

If rules_config_files.txt exists in hayabusa root folder load config from rules_config_files.txt
If rules_config_files.txt does not exist, it works as before.
rules_config_files.txt's content

The following files are combined into one file
channel_abbreviations.txt
channel_eid_info.txt
default_details.txt
event_id_info.txt
eventkey_alias.txt
exclude_rules.txt
generic_abbreviations.txt
level_tuning.txt
noisy_rules.txt
pivot_keywords.txt
proven_rules.txt
provider_abbreviations.txt
target_event_IDs.txt
data_mapping/*.yaml
geoip_field_mapping.yaml

update-rules

If rules_config_files.txt exists in hayabusa root folder, download txt from https://raw.githubusercontent.com/Yamato-Security/hayabusa-encoded-rules/refs/heads/main/rules_config_files.txt
If rules_config_files.txt does not exist, it works as before.

Limitations

level-tuning/set-default-profile commands are not supported.
- User must manually edit rules_config_files.txt

Test

Integration-Test

All commands complited successfully. https://github.com/Yamato-Security/hayabusa/actions/runs/11186632634

CSV timeline and JSON timeline Diff(when rule/config folder exists)

No difference(csv/json) from main branch's results as follows. https://github.com/Yamato-Security/hayabusa/actions/runs/11186634491

I would appreciate it if you could check it out when you have time🙏

rules_config_files.txt exists(and rule/config not exists)

% ls -la
total 36072
drwx------@  6 fukusuke  staff       192 10  5 05:17 .
drwxr-xr-x  17 fukusuke  staff       544 10  5 03:15 ..
-rw-r--r--@  1 fukusuke  staff   7202018 10  5 04:56 encoded_rules.yml
-rwxr-xr-x@  1 fukusuke  staff  10860552 10  5 04:56 hayabusa
drwxr-xr-x@  3 fukusuke  staff        96 10  5 03:16 logs
-rw-r--r--@  1 fukusuke  staff    164589 10  5 04:56 rules_config_files.txt

csv-timeline

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -D -n -u -w -o timeline.csv -q -C
Start time: 2024/10/05 05:20

Total event log files: 585
Total file size: 137.2 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12

Deprecated rules: 216 (4.72%)
Experimental rules: 433 (9.45%)
Stable rules: 255 (5.57%)
Test rules: 3,631 (79.28%)
Unsupported rules: 45 (0.98%)

Hayabusa rules: 181
Sigma rules: 4,399
Total detection rules: 4,580

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 575
Detection rules enabled after channel filter: 4,509

Output profile: standard

Scanning in progress. Please wait.
...
Results Summary:

Events with hits / Total events: 21,137 / 46,413 (Data reduction: 25,276 events (54.46%))

Total | Unique detections: 34,647 | 743
Total | Unique critical detections: 53 (0.15%) | 21 (0.00%)
Total | Unique high detections: 5,768 (16.65%) | 284 (9.29%)
Total | Unique medium detections: 2,461 (7.10%) | 265 (14.00%)
Total | Unique low detections: 6,667 (19.24%) | 104 (35.67%)
Total | Unique informational detections: 19,698 (56.85%) | 69 (38.22%)
...
Saved file: timeline.csv (33.5 MB)
...

json-timeline

% ./hayabusa json-timeline -d ../hayabusa-sample-evtx -D -n -u -w -o timeline.json -q -C

Start time: 2024/10/05 05:21

Total event log files: 585
Total file size: 137.2 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12

Deprecated rules: 216 (4.72%)
Experimental rules: 433 (9.45%)
Stable rules: 255 (5.57%)
Test rules: 3,631 (79.28%)
Unsupported rules: 45 (0.98%)

Hayabusa rules: 181
Sigma rules: 4,399
Total detection rules: 4,580

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 575
Detection rules enabled after channel filter: 4,509

Output profile: standard

Scanning in progress. Please wait.
...
Results Summary:

Events with hits / Total events: 21,137 / 46,413 (Data reduction: 25,276 events (54.46%))

Total | Unique detections: 34,647 | 743
Total | Unique critical detections: 53 (0.15%) | 21 (0.00%)
Total | Unique high detections: 5,768 (16.65%) | 284 (9.29%)
Total | Unique medium detections: 2,461 (7.10%) | 265 (14.00%)
Total | Unique low detections: 6,667 (19.24%) | 104 (35.67%)
Total | Unique informational detections: 19,698 (56.85%) | 69 (38.22%)
...
Saved file: timeline.json (41.0 MB)

update-rules

% ./hayabusa update-rules

...

Start time: 2024/10/05 05:22

Rules file encoded_rules.yml updated successfully.
Config file rules_config_files.txt updated successfully.

eid-metrics

% ./hayabusa eid-metrics -d ../hayabusa-sample-evtx -q -o eid.csv
Generating Event ID Metrics

Start time: 2024/10/05 05:23

Total event log files: 585
Total file size: 137.2 MB

Currently scanning for event ID metrics. Please wait.

[00:00:00] 585 / 585   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Total Event Records: 47,476

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2023-04-14 17:25:12.223 +09:00

Saved results: eid.csv (13.1 KB)

computer-metrics

% ./hayabusa computer-metrics -d ../hayabusa-sample-evtx -q -o cid.csv
Start time: 2024/10/05 05:23

Total event log files: 585
Total file size: 137.2 MB

Currently scanning for computer metrics. Please wait.

[00:00:00] 585 / 585   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Total computers: 65
Saved results: cid.csv (1.5 KB)

Elapsed time: 00:00:00.514

search

% ./hayabusa search -k mimikatz  -d ../hayabusa-sample-evtx -q -o search.csv
Searching...

Start time: 2024/10/05 05:24

Total event log files: 585
Total file size: 137.2 MB

Currently searching. Please wait.

[00:00:00] 585 / 585   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Total findings: 35
Saved results: search.csv (25.8 KB)
Elapsed time: 00:00:00.631

pivot-keywords-list

 % ./hayabusa pivot-keywords-list -d ../hayabusa-sample-evtx -o key -q -w -C
Start time: 2024/10/05 05:34

Total event log files: 585
Total file size: 137.2 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12 (Disabled)

Deprecated rules: 216 (5.02%) (Disabled)
Experimental rules: 432 (10.03%)
Stable rules: 244 (5.67%)
Test rules: 3,631 (84.30%)
Unsupported rules: 45 (1.04%) (Disabled)

Hayabusa rules: 169
Sigma rules: 4,138
Total detection rules: 4,307

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 575
Detection rules enabled after channel filter: 4,239

Output profile: standard

Scanning in progress. Please wait.

[00:00:06] 575 / 575   [========================================] 100%

Scanning finished. Please wait while the results are being saved.
                                                                                                                                               Pivot keyword results were saved to the following files:
key-Source Computers.txt
key-Subject Users.txt
key-Target Users.txt
key-Users.txt
key-Subject Logon IDs.txt
key-Target Logon IDs.txt
key-Logon IDs.txt
key-IP Addresses.txt
key-Source IP Addresses.txt
key-Target IP Addresses.txt
key-Processes.txt
key-Command Lines.txt

Elapsed time: 00:00:07.2074

logon-summary

% ./hayabusa logon-summary -d ../hayabusa-sample-evtx -C -q -o sum.csv
Generating Logon Summary

Start time: 2024/10/05 05:37

Total event log files: 585
Total file size: 137.2 MB

Currently scanning for the logon summary. Please wait.

[00:00:00] 585 / 585   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Total Event Records: 47,476

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2023-04-14 17:25:12.223 +09:00

Successful logon results: sum.csv-successful.csv (6.7 KB)
Failed logon results: sum.csv-failed.csv (274.5 KB)

rules_config_files.txt exists(and rule/config not exists)

level-tuning

% ./hayabusa level-tuning -f rules_config_files.txt -q
Start time: 2024/10/05 05:40

[ERROR] Failed to read level tuning file. path: level_tuning.txt is not correct id format, fix it.

set-defaultprofile

% ./hayabusa set-default-profile -p super-verbose -q
Start time: 2024/10/05 05:41

Default profile cannot be set due to the absence of a config folder. Please check the config folder.

rules_config_files.txt exists(Windows11)

C:\tmp\hayabusa-2.17.0-win-x64-embedded-config>dir
 ドライブ C のボリューム ラベルは Windows です
 ボリューム シリアル番号は 2431-BF8C です

 C:\tmp\hayabusa-2.17.0-win-x64-embedded-config のディレクトリ

2024/10/05  09:35    <DIR>          .
2024/10/05  09:24    <DIR>          ..
2024/10/05  09:32         7,202,018 encoded_rules.yml
2024/10/05  09:30        10,747,904 hayabusa.exe
2024/09/16  13:21    <DIR>          logs
2024/10/05  09:32           164,589 rules_config_files.txt
2024/10/05  09:33        29,012,024 timeline.csv
               4 個のファイル          47,126,535 バイト
               3 個のディレクトリ  289,645,281,280 バイトの空き領域

C:\tmp\hayabusa-2.17.0-win-x64-embedded-config>hayabusa.exe update-rules

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/10/05 09:32

Rules file encoded_rules.yml updated successfully.
Config file rules_config_files.txt updated successfully.

C:\tmp\hayabusa-2.17.0-win-x64-embedded-config>hayabusa.exe csv-timeline -l -w -D -n -u -q -C -o timeline.csv
Start time: 2024/10/05 09:33

Total event log files: 356
Total file size: 250.2 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12

Deprecated rules: 216 (4.72%)
Experimental rules: 433 (9.45%)
Stable rules: 255 (5.57%)
Test rules: 3,631 (79.28%)
Unsupported rules: 45 (0.98%)

Hayabusa rules: 181
Sigma rules: 4,399
Total detection rules: 4,580

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 26
Detection rules enabled after channel filter: 3,880
...
Events with hits / Total events: 28,718 / 122,603 (Data reduction: 93,885 events (76.58%))

Total | Unique detections: 29,473 | 81
Total | Unique critical detections: 381 (1.29%) | 3 (0.00%)
Total | Unique high detections: 223 (0.76%) | 10 (40.74%)
Total | Unique medium detections: 718 (2.44%) | 19 (19.75%)
Total | Unique low detections: 23,790 (80.72%) | 16 (23.46%)
Total | Unique informational detections: 4,361 (14.80%) | 33 (12.35%)

Dates with most total detections:
critical: 2024-06-01 (46), high: 2024-06-04 (24), medium: 2024-07-27 (173), low: 2024-09-28 (3,886), informational: 2024-09-28 (585)

Top 5 computers with most unique detections:
critical: mouse (3)
high: mouse (10)
medium: mouse (18), MyComputer (2)
low: mouse (16)
informational: mouse (33), MyComputer (1), DESKTOP-CNG7416 (1), DESKTOP-9HFNL0J (1)

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                            Top high alerts:                                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Defender Alert (Severe) (372)                                   Antivirus Relevant File Paths Alerts (158)                                      │
│ Antivirus Password Dumper Detection (8)                         Microsoft Defender Blocked from Loading Unsigned DLL (24)                       │
│ Antivirus Exploitation Framework Detection (1)                  Antivirus Hacktool Detection (9)                                                │
│ n/a                                                             Powershell Token Obfuscation - Powershell (8)                                   │
│ n/a                                                             Defender Alert (High) (7)                                                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                              Top low alerts:                                                                 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Potentially Malicious PwSh (543)                                Credential Manager Enumerated (22,575)                                          │
│ Uncommon PowerShell Hosts (83)                                  Credential Manager Accessed (327)                                               │
│ Suspicious Non PowerShell WSMAN COM Provider (24)               Rare Service Installations (319)                                                │
│ BITS Transfer Job With Uncommon Or Suspicious Remote TLD (22)   CodeIntegrity - Unmet Signing Level Requirements By File Under Validation (299) │
│ Usage Of Web Request Commands And Cmdlets - ScriptBlock (11)    Volume Shadow Copy Mount (83)                                                   │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                                                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Logon (Service) (Noisy) (1,374)                                 RDS Sess Start (Noisy) (194)                                                    │
│ Svc Installed (322)                                             RDS Sess Logon (194)                                                            │
│ PwSh Engine Started (318)                                       RDS Sess Logoff (189)                                                           │
│ WMI Provider Started (309)                                      Office App PopUp (180)                                                          │
│ Bits Job Created (216)                                          RDS Sess Disconnect (133)                                                       │
╰───────────────────────────────────────────────────────────────╌─────────────────────────────────────────────────────────────────────────────────╯

Saved file: timeline.csv (29.0 MB)

Elapsed time: 00:00:09.912
...
C:\tmp\hayabusa-2.17.0-win-x64-embedded-config>

rules_config_files.txt not exists(Windows11)

C:\tmp\hayabusa-2.17.0-win-x64-embedded-config>dir rules
 ドライブ C のボリューム ラベルは Windows です
 ボリューム シリアル番号は 2431-BF8C です

 C:\tmp\hayabusa-2.17.0-win-x64-embedded-config\rules のディレクトリ

2024/10/05  09:37    <DIR>          .
2024/10/05  09:37    <DIR>          ..
2024/09/28  17:21    <DIR>          .git
2024/09/03  05:07                28 .gitignore
2024/09/03  05:07             4,394 CHANGELOG-Japanese.md
2024/09/03  05:07             3,740 CHANGELOG.md
2024/10/05  09:32    <DIR>          config
2024/09/28  17:21    <DIR>          doc
2024/09/28  17:21    <DIR>          hayabusa
2024/09/03  05:07        10,564,608 hayabusa-2.17.0-win-x64.exe
2024/09/03  05:07             1,912 LICENSE.md
2024/09/03  05:07            43,124 README-Japanese.md
2024/09/03  05:07            38,278 README.md
2024/09/28  17:21    <DIR>          sigma
               7 個のファイル          10,656,084 バイト
               7 個のディレクトリ  289,682,571,264 バイトの空き領域

C:\tmp\hayabusa-2.17.0-win-x64-embedded-config>hayabusa.exe update-rules -q
Start time: 2024/10/05 09:39

 - Potential File Download Via MS-AppInstaller Protocol Handler (Modified: 2023-11-09 | Path: rules\sigma\sysmon\process_creation\proc_creation_win_susp_ms_appinstaller_download.yml)
 - HackTool - CrackMapExec PowerShell Obfuscation (Modified: 2023-02-21 | Path: rules\sigma\builtin\process_creation\proc_creation_win_hktl_crackmapexec_powershell_obfuscation.yml)
 - PowerShell DownloadFile (Modified: 2021-11-27 | Path: rules\sigma\builtin\process_creation\proc_creation_win_powershell_susp_ps_downloadfile.yml)
 ...

Updated Hayabusa rules: 6
Updated Sigma rules: 1815
Rules updated successfully.

C:\tmp\hayabusa-2.17.0-win-x64-embedded-config>hayabusa.exe csv-timeline -l -D -n -u -w -q -C -o timeline.csv
Start time: 2024/10/05 09:40

Total event log files: 356
Total file size: 250.2 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12

Deprecated rules: 216 (4.72%)
Experimental rules: 431 (9.42%)
Stable rules: 255 (5.57%)
Test rules: 3,627 (79.30%)
Unsupported rules: 45 (0.98%)

Hayabusa rules: 181
Sigma rules: 4,393
Total detection rules: 4,574

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 26
Detection rules enabled after channel filter: 2,282
...
Results Summary:

Events with hits / Total events: 28,779 / 122,673 (Data reduction: 93,894 events (76.54%))

Total | Unique detections: 29,535 | 81
Total | Unique critical detections: 381 (1.29%) | 3 (0.00%)
Total | Unique high detections: 223 (0.76%) | 10 (40.74%)
Total | Unique medium detections: 718 (2.43%) | 19 (19.75%)
Total | Unique low detections: 23,848 (80.74%) | 16 (23.46%)
Total | Unique informational detections: 4,365 (14.78%) | 33 (12.35%)

Dates with most total detections:
critical: 2024-06-01 (46), high: 2024-06-04 (24), medium: 2024-07-27 (173), low: 2024-09-28 (3,886), informational: 2024-09-28 (585)

Top 5 computers with most unique detections:
critical: mouse (3)
high: mouse (10)
medium: mouse (18), MyComputer (2)
low: mouse (16)
informational: mouse (33), MyComputer (1), DESKTOP-CNG7416 (1), DESKTOP-9HFNL0J (1)
Saved file: timeline.csv (29.0 MB)

Elapsed time: 00:00:06.1104
...

rules_config_files.txt not exists and -r option

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -r test.yml -w -q -C -o timeline.csv
Start time: 2024/10/05 10:00

Total event log files: 585
Total file size: 137.2 MB

Loading detection rules. Please wait.

Stable rules: 1 (100.00%)

Hayabusa rules: 1
Total detection rules: 1

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 241
Detection rules enabled after channel filter: 1

Output profile: standard

Scanning in progress. Please wait.

[00:00:00] 241 / 241   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭─────────────────╮
│ Zach Mathis (1) │
╰─────────────────╯

Results Summary:

Events with hits / Total events: 2 / 26,341 (Data reduction: 26,339 events (99.99%))

Total | Unique detections: 2 | 1
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 0 (0.00%) | 0 (100.00%)
Total | Unique medium detections: 0 (0.00%) | 0 (0.00%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 2 (100.00%) | 1 (0.00%)

Dates with most total detections:
critical: n/a, high: n/a, medium: n/a, low: n/a, informational: 2019-02-14 (2)

Top 5 computers with most unique detections:
critical: n/a
high: n/a
medium: n/a
low: n/a
informational: PC01.example.corp (1), PC02.example.corp (1)

╭──────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                    Top high alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                      Top low alerts:  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Logon (RemoteInteractive (RDP)) *Creds in memory* (2)   n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
│ n/a                                                     n/a              │
╰───────────────────────────────────────────────────────╌──────────────────╯

Saved file: timeline.csv (1.5 KB)

rules_config_files.txt not exists and -c option

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -c ../hayabusa-2.17.0-mac-arm/rules/config -w -q -C -o timeline.csv
Start time: 2024/10/05 10:02

Total event log files: 585
Total file size: 137.2 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12 (Disabled)

Deprecated rules: 216 (5.02%) (Disabled)
Experimental rules: 432 (10.03%)
Stable rules: 244 (5.67%)
Test rules: 3,631 (84.30%)
Unsupported rules: 45 (1.04%) (Disabled)

Hayabusa rules: 169
Sigma rules: 4,138
Total detection rules: 4,307

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 575
Detection rules enabled after channel filter: 4,239

Output profile: standard

Scanning in progress. Please wait.

[00:00:07] 575 / 575   [========================================] 100%

when rules directory and encoded_rules.yml exists

% ls -la
total 36008
drwx------@  7 fukusuke  staff       224 10  5 10:14 .
drwxr-xr-x  18 fukusuke  staff       576 10  5 05:50 ..
-rw-r--r--@  1 fukusuke  staff   7202018 10  5 05:22 encoded_rules.yml
-rwxr-xr-x@  1 fukusuke  staff  10860552 10  5 10:15 hayabusa
drwxr-xr-x@  5 fukusuke  staff       160 10  5 05:28 logs
drwxr-xr-x   2 fukusuke  staff        64 10  5 10:13 rules
-rw-r--r--@  1 fukusuke  staff    164589 10  5 05:39 rules_config_files.txt
fukusuke@fukusukenoMacBook-Air hayabusa-2.17.0-mac-arm-encoded % ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -o timeline.csv -C -q
You have the rules directory and encoded_rules.yml in your path. Please delete one of them.

@YamatoSecurity Thank you so much for checking! Yes It's ok to merge!!

Yamato-Security / hayabusa

feat: load config files from a single file #1423

What Changed

Specification

loading config logic

rules_config_files.txt's content

update-rules

Limitations

Test

Integration-Test

CSV timeline and JSON timeline Diff(when rule/config folder exists)

rules_config_files.txt exists(and rule/config not exists)

csv-timeline

json-timeline

update-rules

eid-metrics

computer-metrics

search

pivot-keywords-list

logon-summary

rules_config_files.txt exists(and rule/config not exists)

level-tuning

set-defaultprofile

rules_config_files.txt exists(Windows11)

rules_config_files.txt not exists(Windows11)

rules_config_files.txt not exists and -r option

rules_config_files.txt not exists and -c option

when rules directory and encoded_rules.yml exists