feat: `utf16/utf16be/utf16le/wide` modifiers

fukusuket commented 3 days ago

What Changed

Closed #1432
Refactoring:
- Separated FastMatch and base64offset processing because there was too much processing in mathers.rs

Evidence

Integration-Test

https://github.com/Yamato-Security/hayabusa/actions/runs/11977619969

I would appreciate it if you could check it out when you have time🙏

fukusuket commented 3 days ago

UTF-16 LE

fukusuke@MacBookAir base64-utf-detect % ./target/release/base64-utf-detect ~/Hayabusa/hayabusa-sample-evtx
Possible Base64 + UTF-16 LE(powersploit-security.evtx):  ...

author: TEST
date: 2024/11/22
title: "TEST"
id: 9fa273cc-bcb2-4789-85e3-14ca253ac7f4
level: informational
status: test
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4688
    CommandLine|utf16le|base64offset|contains: $Wc=New-ObJecT
  condition: selection

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -r test.yml -p super-verbose
 -q
...
Timestamp · RuleTitle · Level · Computer · Channel · EventID · RuleAuthor · RuleModifiedDate · Status · RecordID · Details · ExtraFieldInfo · MitreTactics · MitreTags · OtherTags · Provider · RuleCreationDate · RuleFile · EvtxFile
2016-09-21 04:15:54.128 +09:00 · TEST · info · IE10Win7 · Sec · 4688 · TEST · - · test · 13488 · Cmdline: powershell.exe -NoP -sta -NonI -W Hidden -Enc <base64> ¦ Proc: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe ¦ PID: 3688 ¦ User: IEUser ¦ LID: 0x6793c · ProcessId: 512 ¦ SubjectDomainName: IE10WIN7 ¦ SubjectUserSid: S-1-5-21-3463664321-2923530833-3546627382-1000 ¦ TokenElevationType: ELEVATED_TOKEN ·  ·  ·  · Sec · 2024/11/22 · test.yml · ../hayabusa-sample-evtx/DeepBlueCLI/powersploit-security.evtx
...
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ TEST (1)                    n/a              │
│ n/a                         n/a              │

fukusuket commented 3 days ago

Wide

fukusuke@MacBookAir base64-utf-detect % ./target/release/base64-utf-detect ~/Hayabusa/hayabusa-sample-evtx
Possible Base64 + UTF-16 LE(powersploit-security.evtx):  ...

author: TEST
date: 2024/11/22
title: "TEST"
id: 9fa273cc-bcb2-4789-85e3-14ca253ac7f4
level: informational
status: test
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4688
    CommandLine|wide|base64offset|contains: $Wc=New-ObJecT
  condition: selection

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -r test.yml -p super-verbose -q
...
Timestamp · RuleTitle · Level · Computer · Channel · EventID · RuleAuthor · RuleModifiedDate · Status · RecordID · Details · ExtraFieldInfo · MitreTactics · MitreTags · OtherTags · Provider · RuleCreationDate · RuleFile · EvtxFile
2016-09-21 04:15:54.128 +09:00 · TEST · info · IE10Win7 · Sec · 4688 · TEST · - · test · 13488 · Cmdline: powershell.exe -NoP -sta -NonI -W Hidden -Enc <base64> ¦ Proc: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe ¦ PID: 3688 ¦ User: IEUser ¦ LID: 0x6793c · ProcessId: 512 ¦ SubjectDomainName: IE10WIN7 ¦ SubjectUserSid: S-1-5-21-3463664321-2923530833-3546627382-1000 ¦ TokenElevationType: ELEVATED_TOKEN ·  ·  ·  · Sec · 2024/11/22 · test.yml · ../hayabusa-sample-evtx/DeepBlueCLI/powersploit-security.evtx
...
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ TEST (1)                    n/a              │

fukusuket commented 3 days ago

UTF-16

fukusuke@MacBookAir base64-utf-detect % ./target/release/base64-utf-detect ~/Hayabusa/hayabusa-sample-evtx
Possible Base64 + UTF-16 LE(powersploit-security.evtx):  ...

author: TEST
date: 2024/11/22
title: "TEST"
id: 9fa273cc-bcb2-4789-85e3-14ca253ac7f4
level: informational
status: test
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4688
    CommandLine|utf16|base64offset|contains: $Wc=New-ObJecT
  condition: selection

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -r test.yml -p super-verbose -q
...
Timestamp · RuleTitle · Level · Computer · Channel · EventID · RuleAuthor · RuleModifiedDate · Status · RecordID · Details · ExtraFieldInfo · MitreTactics · MitreTags · OtherTags · Provider · RuleCreationDate · RuleFile · EvtxFile
2016-09-21 04:15:54.128 +09:00 · TEST · info · IE10Win7 · Sec · 4688 · TEST · - · test · 13488 · Cmdline: powershell.exe -NoP -sta -NonI -W Hidden -Enc <base64> ¦ Proc: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe ¦ PID: 3688 ¦ User: IEUser ¦ LID: 0x6793c · ProcessId: 512 ¦ SubjectDomainName: IE10WIN7 ¦ SubjectUserSid: S-1-5-21-3463664321-2923530833-3546627382-1000 ¦ TokenElevationType: ELEVATED_TOKEN ·  ·  ·  · Sec · 2024/11/22 · test.yml · ../hayabusa-sample-evtx/DeepBlueCLI/powersploit-security.evtx
...
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ TEST (1)                    n/a              │
│ n/a                         n/a              │
...

fukusuket commented 3 days ago

base64offset

fukusuke@MacBookAir base64-utf-detect % ./target/release/base64-utf-detect ~/Hayabusa/hayabusa-sample-evtx
...
Possible Base64 + UTF-8("susp_explorer_exec.evtx"): $XX=IEX( ...

author: TEST
date: 2024/11/22
title: "TEST"
id: 9fa273cc-bcb2-4789-85e3-14ca253ac7f4
level: informational
status: test
logsource:
  product: windows
  service: sysmon
detection:
  selection:
    Channel: Microsoft-Windows-Sysmon/Operational
    EventID: 1
    CommandLine|base64offset|contains: $XX=IEX
  condition: selection

./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -r test.yml -p super-verbose -q
...
Timestamp · RuleTitle · Level · Computer · Channel · EventID · RuleAuthor · RuleModifiedDate · Status · RecordID · Details · ExtraFieldInfo · MitreTactics · MitreTags · OtherTags · Provider · RuleCreationDate · RuleFile · EvtxFile
2019-08-14 21:17:14.893 +09:00 · TEST · info · MSEDGEWIN10 · Sysmon · 1 · TEST · - · test · 10675 · Cmdline: "c:\windows\system32\wscript.exe" /E:vbs c:\windows\temp\icon.ico "powershell -exec bypass -c <...> ¦ Proc: C:\Windows\System32\wscript.exe ¦ User: MSEDGEWIN10\IEUser ¦ ParentCmdline: "C:\Windows\system32\rundll32.exe" zipfldr.dll,RouteTheCall shell:::{769f9427-3cc6-4b62-be14-2a705115b7ab} ¦ LID: 0x29126 ¦ LGUID: 747F3D96-F419-5D53-0000-002026910200 ¦ PID: 2876 ¦ PGUID: 747F3D96-FBCA-5D53-0000-001036784100 ¦ ParentPID: 2476 ¦ ParentPGUID: 747F3D96-FBCA-5D53-0000-0010B8664100 ¦ Description: Microsoft ® Windows Based Script Host ¦ Product: Microsoft ® Windows Script Host ¦ Company: Microsoft Corporation ¦ Hashes: SHA1=267D05CE8D10D97620BE1C7773757668BAEB19EE,MD5=F5E5DF6C9D62F4E940B334954A2046FC,SHA256=47CACD60D91441137D055184614B1A418C0457992977857A76CA05C75BBC1B56,IMPHASH=0F71D5F6F4CBB935CE1B09754102419C · CurrentDirectory: C:\Windows\system32\ ¦ FileVersion: 5.812.10240.16384 ¦ IntegrityLevel: Medium ¦ ParentImage: C:\Windows\System32\rundll32.exe ¦ RuleName:  ¦ TerminalSessionId: 1 ¦ UtcTime: 2019-08-14 12:17:14.661 ·  ·  ·  · Sysmon · 2024/11/22 · test.yml · ../hayabusa-sample-evtx/EVTX-ATTACK-SAMPLES/Execution/sysmon_lolbas_rundll32_zipfldr_routethecall_shell.evtx
...
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ TEST (2)                    n/a              │

YamatoSecurity commented 2 days ago

@fukusuket Thanks so much! Did you also implement utf16be?

I tried changing le to be here:

author: TEST
date: 2024/11/22
title: "TEST"
id: 9fa273cc-bcb2-4789-85e3-14ca253ac7f4
level: informational
status: test
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4688
    CommandLine|utf16be|base64offset|contains: $Wc=New-ObJecT
  condition: selection

and it still detected it, which I don't think it should because the bit order is reversed. Can you check this?

fukusuket commented 2 days ago

@YamatoSecurity Thank you so much for checking!
Yes, I had implemented the process for UTF-16BE! I think the above behavior is as expected.🤔

I created simple program and checked the byte sequence and it is a partial match for both UTF-16 BE/UTF-16 LE as follows. (It does not match UTF-8 sequences). https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=a7f22e3dd74061d8b6c313530a930cf3

Original: "$Wc=New-ObJecT"
UTF-8     encoded bytes [36, 87, 99, 61, 78, 101, 119, 45, 79, 98, 74, 101, 99, 84]
UTF-16 be encoded bytes [0, 36, 0, 87, 0, 99, 0, 61, 0, 78, 0, 101, 0, 119, 0, 45, 0, 79, 0, 98, 0, 74, 0, 101, 0, 99, 0, 84, ]
UTF-16 le encoded bytes [36, 0, 87, 0, 99, 0, 61, 0, 78, 0, 101, 0, 119, 0, 45, 0, 79, 0, 98, 0, 74, 0, 101, 0, 99, 0, 84, 0, ]

In https://github.com/Yamato-Security/hayabusa/pull/1503#issuecomment-2495737894 case, the actual is the same logic as the following rule.

author: TEST
date: 2024/11/22
title: "TEST"
id: 9fa273cc-bcb2-4789-85e3-14ca253ac7f4
level: informational
status: test
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4688
  selection_utf_16_le:  # same as utf16le|base64offset|contains: $Wc=New-ObJecT
    CommandLine|contains: 
      - JABXAGMAPQBOAGUAdwAtAE8AYgBKAGUAYwBUA
      - QAVwBjAD0ATgBlAHcALQBPAGIASgBlAGMAVA # matched
      - kAFcAYwA9AE4AZQB3AC0ATwBiAEoAZQBjAFQA
  selection_utf_16_be:  # same as utf16be|base64offset|contains: $Wc=New-ObJecT
    CommandLine|contains: 
      - ACQAVwBjAD0ATgBlAHcALQBPAGIASgBlAGMAV # matched
      - AkAFcAYwA9AE4AZQB3AC0ATwBiAEoAZQBjAF
      - AJABXAGMAPQBOAGUAdwAtAE8AYgBKAGUAYwBU
  condition: selection and  1 of selection_*

In the above rule, it matches both selection_utf_16_le and selection_utf_16_be, so the same log is detected for both utf16be|base64offset and utf16le|base64offset

Yamato-Security / hayabusa