search

jasonish / py-idstools

idstools: Snort and Suricata Rule and Event Utilities in Python (Including a Rule Update Tool)
Other
277 stars 85 forks source link

memory usage increase issue #92

Open ganduulgag opened 2 years ago

ganduulgag commented 2 years ago

hi,

is there someone who experienced a memory usage increase issue when you run rulecat? if yes, how did you tackle the issue?

in our case, we configured rulecat to run once a day in cron job as we use Security Onion. Whenever the cron job starts, memory usage increases drastically. We suppressed several rules with idstools. As a solution, we added swap space for the moment but we are concerned if this memory usage keeps increasing but we don't want to allocate more swap or memory.

jasonish commented 2 years ago

I just looked at rulecat with the ET/Open ruleset and it did use about 500MB. Not ideal, but it does load, parse and sort all the rules in memory. But this might not be the only thing using memory. rulecat can be configured to test the output with suricata -T. This causes suricata to load the rules in memory to check for sanity which can use even more memory. This is somewhat of a known issue, but not much can be done about it if you want to validate the rules before restarted Suricata.

ganduulgag commented 2 years ago

where can I configure to test output with suricata -T? Currently, I am using rulecat with ETPro rules and when I run the rulecat, memory usage increases about 3GB which is quite a lot memory for the system. is there a way I can reduce memory usage? besides, it takes roughly 3 min to finish the rulecat. Here is my console output after running rulecat:

2022-09-07 04:05:02,669 - -- Loading ./rulecat.conf. 2022-09-07 04:05:02,682 - -- Forcing Suricata version to 6.0. 2022-09-07 04:05:02,695 - -- Fetching https://urlhaus.abuse.ch/downloads/urlhaus_suricata.tar.gz. 100% - 741376/741376 2022-09-07 04:05:02,934 - -- Done. 2022-09-07 04:05:03,126 - -- Fetching https://rules.emergingthreatspro.com/0148694801847852/suricata-6.0.0/etpro.rules.tar.gz. 100% - 8922965/8922965 2022-09-07 04:05:04,836 - -- Done. 2022-09-07 04:05:05,259 - -- Fetching https://sslbl.abuse.ch/blacklist/sslipblacklist.rules. 100% - 14294/14294 2022-09-07 04:05:05,401 - -- Done. 2022-09-07 04:05:05,403 - -- Fetching https://threatfox.abuse.ch/downloads/threatfox_suricata.tar.gz. 100% - 1073155/1073155 2022-09-07 04:05:05,908 - -- Done. 2022-09-07 04:05:06,098 - -- Ignoring file rules/deleted.rules 2022-09-07 04:05:57,472 - -- Loaded 155162 rules. 2022-09-07 04:08:32,612 - -- Disabled 1233 rules. 2022-09-07 04:08:32,612 - -- Enabled 0 rules. 2022-09-07 04:08:32,612 - -- Modified 19831 rules. 2022-09-07 04:08:32,612 - -- Dropped 0 rules. 2022-09-07 04:08:35,223 - -- Enabled 184 rules for flowbit dependencies. 2022-09-07 04:09:10,379 - -- Writing rules to /home/kali/all.rules: total: 155162; enabled: 137950; added: 13435; removed 11563; modified: 1359 2022-09-07 04:09:13,878 - -- Done

jasonish commented 2 years ago

There is a command line option, --test-command that you can then provide a command to run the test. Something like --test-command "suricata -T" might work if using all the defaults.

If you are only using this tool for Suricata, you should really look at suricata-update which is bundled with Suricata these days. Its had some work to reduce memory usage over time, and runs suricata -T by default.