dspruell
commented
11 months ago Resolved ending with commit aa508ef67e33c5361447a83291dc3c614ef67ddd.
Now raise an exception when a macro would render a regex string that would trigger this:
$ grep regexpalt rules/file-characteristics.yml
$s = /{{ url_shortener_domains|regexpalt }}/ nocase ascii wide$ ./env/bin/yara-yaml rules/file-characteristics.yml
2023-12-19 11:45:45,476 [ERROR] filters:151 - macro has too many values (176) and no base string is specified
Traceback (most recent call last):
File "/home/dspruell/devel/yara-yaml/./env/bin/yara-yaml", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/dspruell/devel/yara-yaml/env/lib/python3.11/site-packages/yarayaml/cli.py", line 77, in main
for rule in builder.get_yara_rules():
File "/home/dspruell/devel/yara-yaml/env/lib/python3.11/site-packages/yarayaml/builder.py", line 204, in get_yara_rules
yield self.apply_templating(self.rule_template, r)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dspruell/devel/yara-yaml/env/lib/python3.11/site-packages/yarayaml/builder.py", line 130, in apply_templating
context["rule_strings"] = rule_strings_template.render(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dspruell/devel/yara-yaml/env/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/home/dspruell/devel/yara-yaml/env/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
File "/home/dspruell/devel/yara-yaml/env/lib/python3.11/site-packages/yarayaml/filters.py", line 155, in regexpalt
raise TooManyMacroValuesError(
yarayaml.exceptions.TooManyMacroValuesError: The number of values in a the macro exceeds the amount that may result in a rule that won't function due to YARA regular expression complexity limits. It is recommended to set the filter to use the `strname` argument to place values into multiple strings and then adjust the rule conditions to work with this, if required.When using the alternate filter format, the macro values are split into multiple, smaller strings:
$ grep regexpalt rules/file-characteristics-block.yml
{{ url_shortener_domains|regexpalt(strname="url_shortener_domains", modifiers=["nocase", "ascii", "wide"]) }}$ ./env/bin/yara-yaml rules/file-characteristics-block.yml
import "dotnet"
import "elf"
import "hash"
import "magic"
import "math"
import "pe"
import "time"
rule FC_File_with_URL_Shortener_Reference : url_shortener
{
meta:
EventID = "999999"
Revision = "1"
Category = "File Characteristic"
Description = "Adversaries may abuse URL shortener services to hide malicious content behind a legitimate domain in an attempt by evade detection. This signature identifies files that contain references to known URL shortener service domains."
Author = "Darren Spruell"
Confidence = 9
Severity = 2
Created_Date = "2023-12-12"
Last_Modified_Date = "2023-12-12"
Reference01 = "https://safecomputing.umich.edu/be-aware/phishing-and-suspicious-email/shortened-url-security"
Tag01 = "inquest:intel-type=\"capability-intelligence\""
Tag02 = "inquest:rule-source=\"inquest\""
Tag03 = "inquest:capability=\"url-shortener\""
strings:
$url_shortener_domains_01 = /\b(0\.gp|2\.gp|2\.ly|2no\.co|3\.ly|4\.gp|4\.ly|5\.gp|6\.gp|6\.ly|6url\.ru|7\.ly|8\.ly|9\.ly|a\.gd|adf\.ly|adfoc\.us|adyou\.me|aka\.ms|appurl\.io|aria4c\.com|arm\.in|atln\.cm|ay\.gy|bc\.vc|bcvc\.ink|binged\.it|bit\.do|bit\.ly|bitly\.com|blltly\.com|bltlly\.com|bluenik\.com|bom\.to|brook\.gs|budurl\.com|buff\.ly|by\.tn|byltly\.com|ceesty\.com|chilp\.it|cinurl\.com|clck\.ru|cli\.gs|clk\.im|clkmein\.com|cllkme\.com|corneey\.com|cutt\.ly|cutt\.us|destyy\.com|dlvr\.it|fancli\.com|festyy\.com|fon\.gs|fwd4\.me|g\.asia|g2\.by|ga\.tc|geags\.com|gee\.su|gestyy\.com|gg\.gg|ggle\.io|git\.io|goatcodes\.com|goo\.gl|goo\.su|grabify\.link|gtly\.to|hex\.io|hideuri\.com|hiokurl\.com|hub\.am|hurl\.ws|ift\.tt|ind\.pn|is\.gd|itsssl\.com|j\.gs|j\.mp|jfilte\.com|jinyurl\.com|jnw0\.com|kializer\.com|kl\.am|kutt\.it|l\.ead\.me|l\-k\.one|linkiti\.com|lk\.tc|longurl\.in|me2\.kr|microify\.com|migre\.me|myown\.bio|ne\.fo|onelink\.to|onforb\.es|onx\.la|ouo\.io|ouo\.press|ow\.ly|p\.asia|parg\.co|picfs\.com|pixelfy\.me|pixly\.me|plot\.le|pnut\.co|prosyn\.org|pxlme\.me|q\.gs|qrco\.de|rb\.gy|rebrand\.ly|riffhold\.com|s\.id|s\.net\.vn|serv\.mpaging\.com)\b/ nocase ascii wide
$url_shortener_domains_02 = /\b(sf\.to|sh\.st|short\.ie|short\.to|shorten\-this\.link|shoxet\.com|shrunken\.com|shurll\.com|skamaker\.com|skamason\.com|sn\.im|snip\.ly|snipr\.com|snipurl\.com|snurl\.com|srsp\.app|ssur\.cc|ssurll\.com|suo\.im|t\.cn|t\.co|t\.ly|t2m\.io|teknik\.io|thetim\.es|tinurli\.com|tiny\.cc|tinyl\.in|tinyurl\.com|tiurll\.com|tknk\.io|to\.ly|tr\.im|tweetburner\.com|twurl\.nl|u\.to|ub0\.cc|ur1\.ca|ur3\.us|urlca\.com|urlcod\.com|urlgoal\.com|urllio\.com|urloso\.com|urls\.bz|urluso\.com|urluss\.com|utm\.io|vtaurl\.com|wwdurl\.com|www\.shrunken\.com|x\.co|yamechanic\.com|yoineer\.com|yoitect\.com|zbshort\.live)\b/ nocase ascii wide
condition:
any of them
}$ ./env/bin/yara-yaml rules/file-characteristics-block.yml > tmp.yar
$ yara tmp.yar /dev/null
$ echo $?
0
This can occur when the length of a list macro converted to an alternation regex is long. This was tested with
url_shortener_domains.yml, which has nearly 180 list elements. The error occurs when all values are packed into a single regular expression string in YARA; when the list values are split between two separate regex strings, the error does not occur.Tested in YARA v4.1.3.