Error parsing detections with list of maps

[pathtofile]

Hey mate, super appreciative of this library, I've just discovered it fails to parse some rules from the official Sigma repo, e.g. this one. It erorrs out with cannot unmarshal !!map into string.

I am super new to SIGMA rules, but digging into it, the issue could possibly be that looking at the spec can be lists of maps. E.g. this basic rules fails to be parsed by sigma-go but according the spec it is valid (with the CommandLine being logical OR'd together):

title: Test that doesn't work
id: e0b0c2ab-3d52-46d9-8cb7-049dc775fbd1
status: experimental
description: A Sigma rule that fails to parse using sigma-go
author: pathtofile
date: 2022/01/23
logsource:
    category: process_creation
    product: windows
level: high
detection:
    baddetect:
        - CommandLine|contains: 'apple'
        - CommandLine|contains: 'bannana'
    condition: baddetect

[pathtofile]

Error I think is coming from here: https://github.com/bradleyjkemp/sigma-go/blob/main/rule_parser.go#L88

[bradleyjkemp]

Thanks for raising this, this is a very interesting case!

According to the text specification, a detection can either be:

• A list of string keywords
• A map of filednames to values

This is backed up by the yaml schema:

detection:
    // ... omitted for brevity
    rest:
        type: //any
        of:
            - type: //arr
              contents: //str
            - type: //map
              values:
                  type: //any
                  of:
                      - type: //str
                      - type: //arr
                        contents: //str
                        length:
                            min: 2

However, it's reasonable to argue that sigmac should be the source of truth for what's a valid rule, not the text specification 🙂

Unfortunately, I'm not sure how easily this could be supported in sigma-go. I'm also unsure it's a good feature to support. Your example can trivially be rewritten as:

baddetect:
    CommandLine|contains:
        - 'apple'
        - 'banana'

And more complex examples can be split out into e.g.

baddetect-1:
    CommandLine|contains: 'apple'
baddetect-2:
    CommandLine|contains: 'banana'

with a condition using (any of baddetect-*)

So for now, I'm inclined to intentionally not support this case (unless in future we find a compelling example where a list of maps really is the best way to write a rule)?

[pathtofile]

Hey @bradleyjkemp, no worries, it's your library, and I do agree with your points. From my perspective I view any rules in the official Sigma repo (that are there on purpose and not accedentily broken, etc.) as the source of truth for both how people should/will write Sigma rules, and how people should/will want to consume them.

But I agree there are straightforward workarounds, and I'm only using your (awesome) library in a personal project, so feel free to close this issue :-)

[veramine]

Hey @bradleyjkemp, I've started evaluating your library in an attempt to add sigma search capability to our collection of collected endpoint data. It works great for sample data but I ran into this same issue when testing against real-world sigma rules. This format of detection specification via list of maps seems quite common. Here were the first 50 examples I found where the rules could not be parsed due to this issue. There are hundreds or thousands more from the published sigma rule repo. Any chance we can put in a vote to enable this support? I'd be happy to take a crack at implementing it myself if you are open to supporting it. Thanks!

https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_accesschk_usage_after_priv_escalation.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_always_install_elevated_windows_installer.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_anydesk.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_anydesk_susp_folder.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_bluemashroom.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_chafer_mar18.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_gallium.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_hurricane_panda.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_judgement_panda_gtr19.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_mustangpanda.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_sofacy.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_ta505_dropper.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_apt_wocao.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_attrib_system_susp_paths.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_bad_opsec_sacrificial_processes.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_base64_invoke_susp_cmdlets.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_base64_reflective_assembly_load.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_bootconf_mod.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_bypass_squiblytwo.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_certoc_execution.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_change_default_file_assoc_susp.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_clip.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_cmd_delete.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_cmdkey_recon.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_copying_sensitive_files_with_credential_data.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_creation_mavinject_dll.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_curl_download.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_dinjector.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_dns_exfiltration_tools_execution.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_dns_serverlevelplugindll.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_dsacls_abuse_permissions.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_dsacls_password_spray.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_encoded_frombase64string.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_encoded_iex.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_enumeration_for_credentials_cli.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_enumeration_for_credentials_in_registry.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_esentutl_webcache.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_exfil_data_via_cli.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_fsutil_drive_enumeration.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_gotoopener.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_hack_rubeus.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_hack_wce.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_hacktool_imphashes.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_hktl_createminidump.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_hktl_uacme_uac_bypass.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_icacls_deny.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_impacket_compiled_tools.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_invoke_obfuscation_obfuscated_iex_commandline.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_logmein.yml https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_lolbin_adplus.yml

[bradleyjkemp]

🤔 @veramine would you mind opening a PR in the main Sigma repo to clarify what the semantics are for this?

I assume the conditions would be OR-ed together but I'm not keen to implement this until it's actually in the spec (better to be incomplete than incompatible)

[veramine]

Yep, I'll report back here when spec is clarified, thanks!

[thomaspatzke]

Hi @bradleyjkemp! I'm one of the Sigma maintainers, initiated pySigma/sigmatools and want to clarify this.

Unfortunately, the Sigma specification is indeed inconsistent regarding this feature, the list section only specifies the usage of strings while the section about maps defines the expected behavior:

Lists of maps are joined with a logical 'OR'. All elements of a map are joined with a logical 'AND'.

Sorry for the confusion! I will fix the spec.

The implementation in pySigma was a bit tricky. I solved this as follows:

• A SigmaDetection object represents a detection definition or a map.
• A SigmaDetectionItem object represent a key-value-item in a map or a plain value.
• The list of detection_items in the SigmaDetection class can contain SigmaDetectionItem as well as SigmaDetection objects. Latter is used for list of maps.
• Obviously, both cases must be considered everywhere where detection_items are touched.

Perhaps this helps a bit to implement this in sigma-go.

By the way, I personally don't really like this feature because it adds some complexity to each Sigma implementation. I had the deep desire to drop this completely while writing pySigma and sometimes before. But as @veramine pointed out, the usage is quite prevalent. Rule writers seem to like it and a goal of Sigma is to make rule-writing convenient, so I think it's better to live with this beast 😉 And sometimes it was also useful for implementation of features.

[thomaspatzke]

Just updated the spec, please check if this is now described clearer.

[bradleyjkemp]

@thomaspatzke thanks for the clarification and background on the history of the feature

Rule writers seem to like it and a goal of Sigma is to make rule-writing convenient, so I think it's better to live with this beast 😉

💯 at the end of the day real-world usage is what we're here for

Your Python solution's neat. I think the same technique will work for sigma-go too

[veramine]

You're the BEST! Thanks so much!!! ❤️