williballenthin
commented
1 year ago Hi @yelhamer
Thank you for the very detailed feature request! I think I understand the problem and how you propose to solve it. I don't disagree with the problem, though I would like to discuss further how we can solve it, particularly because you propose to change our rule syntax (something that has the potential to break existing installations, so we want to do this carefully).
First, can you identify multiple scenarios in which you'd want to mix the entropy feature with other logic? In other words, could we achieve the same effect in practice by introducing a new characteristic(high-section-entropy) feature and hardcoding the characteristic feature extraction? The benefit here is that we don't need new syntax and maybe don't need all the proposed complexity. I can think of a couple scenarios, but I'd like to hear what you can come up with.
I'd also propose another alternative implementation: add a new scope for "section features" and introduce section name and entropy features in this scope. We'd also have to implement some sort of range matching, so you can specify things like "greater than 7.0". With this, the rule syntax might look like:
rule:
meta:
name: packed .text section
scope: section
rule:
section-name: .text
entropy: 7.0 or moreor with a subscope block
rule:
meta:
name: something something
scope: file
rule:
# ... other logic at file scope...
section:
- section-name: .text
- entropy: 7.0 or more
# ... other logic at file scope...The benefit here is that we're using our existing infrastructure to handle conjunctions of features, like name and entropy, rather than implying them with a new syntax like [".text", 5.7]. However, this would take a bit more work, since we have to introduce the new scope (still not too hard). Fortunately, I think this scope does make sense, because it it's a concept that applies to many formats, like PE, ELF, MachO, etc. I wonder what other features make sense at a section scope? name, entropy, permissions, size, ...?
Imagine:
- name: high entropy code
- section:
- and:
- permissions: execute
- entropy: 7.0 or more - name: unpacked region in memory
- section:
- and:
- permissions: execute
- physical-size: 0
- virtual-size: 0 or more
yelhamer
mr-tz
mike-hunhoff
Summary
This new feature would compute Shannon's entropy for different sections of the executable, and then compare the computed (section name, entropy) pairs against the section names and the associated threshold values specified by the user in the rule file. If the section's entropy value surpasses the rule-defined threshold, then that feature evaluates to True, otherwise it evaluates to False.
This addition would afford capa the ability to perform entropy analysis for binary files, which in turn would make detecting packed, obfuscated, and encrypted binaries/data more feasible.
Motivation
At its current state, capa relies primarily on api and file signatures as well as instruction mnemonics to detect different binary packing and encryption/obfuscation capabilities; however, this approach could be made more efficient by introducing per-section entropy analysis, since entropy provides a straightforward metric for detecting packed data. This is because high-value entropy is a characteristic of packed/encrypted software [1], with values above 6.8 being generally associated with sections containing mostly encrypted data, and values below 5 being associated with sections that contain no encrypted data. This information could be paired with other features to ascertain whether an executable is packed or not.
For example, suppose we wish to write a rule that detects xor-obfuscation in which the xor operation is implemented using nand gates. Writing a corresponding rule for this capability using the traditional approach would be somewhat difficult, since the key capa features associated with this capability (the api signatures and the mnemonics: and, not) are quite common among non-malicious software; however, if we combine these features with a high entropy threshold value (.text section entropy value of 6.5 for example), then we can get a much clearer indication of whether the input file is packed or not.
Syntax and implementation
I could think of 4 formats — listed below — for implementing this feature, but could not decide which one was best. For the first 3 formats, the section name and the associated entropy would be stored as the value member of the entropy feature, and for the final format they would be stored in two independent features that are children of a Statement resembling the entropy feature.
Here are the formats:
For the first two approaches, I am not quite sure if the addition of tuples/lists values to capa can be done at the moment, and would need the developers say on that.
The third approach makes the most sense to me, since it is the best looking and the easiest to implement; however, storing two separate values in the same string feels somewhat like a hack.
As for the last approach, I don't see any difficulties adding floats to the parse_description() function, as it would require only some _valuetype and isinstance() checks to be added. However, I believe implementing the section-entropy keyword as a Statement would be a huge hack.
GSoC
I wish to be assigned this issue as part of my GSoC proposal. I have prepared some of the code required for the introduction of this feature, and can submit a PR shortly after the developers approve the request and provide their feedback (should they be in favor of adding it).
References
[1] R. Lyda and J. Hamrock, "Using Entropy Analysis to Find Encrypted and Packed Malware," in IEEE Security & Privacy, vol. 5, no. 2, pp. 40-45, March-April 2007, doi: 10.1109/MSP.2007.48. Open-Access Link