Use `load` keyword for runtime dynamic dataset loading

[satta]

Suricata provides the load keyword in a rule definition to dynamically load datasets when referenced. It would be nice if the ruleset used this keyword so one would not have to edit suricata.yaml to preconfigure the dataset; it would just be loaded on demand as long as the .lst file is placed in the same directory as the rule.

AFAICS the lines in phishing.lst should also be given as base64 encoded strings (cf https://docs.suricata.io/en/latest/rules/datasets.html#file-formats)

This would also allow suricata-update (https://github.com/OISF/suricata-update) to handle datasets distributed with the rules.

[julioliraup]

gg, i'll set it

[julioliraup]

It will be necessary to use datarep. You could use it, but I imagine that setting a high reputation for all values would require more processing to check the reputation of each file. As it is, just checking for the existence already ensures the alert.

[satta]

Not sure if we're talking about the same thing here...? I was not referring to reputation data, but to the domains in the first two rules:

alert dns $HOME_NET any -> any any (msg:"AT Related Malicious Domains in DNS Lookup"; dns.query; dataset:isset, phishing-domain; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000000; rev:1; metadata: signature_severity Major, created_et 2024_06_20;)
alert tls $HOME_NET any -> any any (msg:"AT Related Malicious Domains in TLS SNI"; tls.sni; dataset:isset, phishing-domain; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000001; rev:1; metadata: signature_severity Major, created_et 2024_06_20;)

which could be rewritten (after changing the dataset file to base64) to include type string, load phishing.lst which would then remove the need to change suricata.yaml because the filename is already referenced in the rule itself.

[julioliraup]

I understand. According to the documentation, to use "load" it is necessary to define it as an attribute within the datarep field, as shown in the example:

dataset:<set|isset|isnotset>,<name> \
    [, type <string|md5|sha256|ipv4|ip>, save <file name>, load <file name>, state <file name>,

[satta]

Both datarep and dataset have the additional load keyword, and here only the dataset case is relevant. You even mention dataset and not datarep in your reply above ;)

What I meant was making this change:

diff --git a/antiphishing.rules b/antiphishing.rules
index 1169ee4..139130f 100644
--- a/antiphishing.rules
+++ b/antiphishing.rules
@@ -6,8 +6,8 @@
 # This rule updates hourly 
 #

-alert dns $HOME_NET any -> any any (msg:"AT Related Malicious Domains in DNS Lookup"; dns.query; dataset:isset, phishing-domain; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000000; rev:1; metadata: signature_severity Major, created_et 2024_06_20;)
-alert tls $HOME_NET any -> any any (msg:"AT Related Malicious Domains in TLS SNI"; tls.sni; dataset:isset, phishing-domain; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000001; rev:1; metadata: signature_severity Major, created_et 2024_06_20;)
+alert dns $HOME_NET any -> any any (msg:"AT Related Malicious Domains in DNS Lookup"; dns.query; dataset:isset, phishing-domain, type string, load phishing-base64.txt; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000000; rev:1; metadata: signature_severity Major, created_et 2024_06_20;)
+alert tls $HOME_NET any -> any any (msg:"AT Related Malicious Domains in TLS SNI"; tls.sni; dataset:isset, phishing-domain, type string, load phishing-base64.txt; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000001; rev:1; metadata: signature_severity Major, created_et 2024_06_20;)
 alert http $HOME_NET any -> any any (msg:"AT related malicious URL (www .trx016 .asia/)"; flow:established,to_server; http.method; content:"GET"; http.uri; content:"/a"; startswith; fast_pattern; http.host; bsize:16; content:"www.trx016.asia"; reference:url,phishstats.info; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000002; rev:1; metadata: signature_severity Major, created_et 2024_06_22;)
 alert http $HOME_NET any -> any any (msg:"AT related malicious URL (cricketbettingpro .com/home .html/autoload .ht)"; flow:established,to_server; http.method; content:"GET"; http.uri; content:"/home.html/autoload.htm"; startswith; fast_pattern; http.host; bsize:22; content:"cricketbettingpro.com"; reference:url,phishstats.info; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000005; rev:1; metadata: signature_severity Major, created_et 2024_06_22;)
 alert http $HOME_NET any -> any any (msg:"AT related malicious URL (tridipbarman09 .github .io/netflix-clon)"; flow:established,to_server; http.method; content:"GET"; http.uri; content:"/netflix-clone"; startswith; fast_pattern; http.host; bsize:25; content:"tridipbarman09.github.io"; reference:url,phishstats.info; reference:url,github.com/julioliraup/Antiphishing; classtype:social-engineering; sid:6000006; rev:1; metadata: signature_severity Major, created_et 2024_06_22;)

and also converting the .lst to base64:

$ for l in `cat phishing.lst`; do echo "$l" | base64 -w2000 ; done > phishing-base64.txt

because with phishing.lst you would otherwise get:

Error: datasets: bad base64 encoding phishing-domain//home/satta/tmp/Antiphishing/phishing.lst [DatasetLoadString:datasets.c:528]

when trying to load.

[julioliraup]

Sorry hahaha I'see now hahah

[julioliraup]

I'll change it, migrate phishing.lst for base64 default. U can make PR?

[satta]

Thanks for taking care of this!