Writing to multiple rules files results in duplicate rules

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run pulledpork as normal with the '-k' flag.
2. Immediately run pulledpork using '-k -P' flags.

What is the expected output? What do you see instead?
The expectation is that the output directory will be identical after each run. 
Instead a duplicate entry is added for each rule. 

For example, my config includes both the VRT and the ET sets. If I run an 
update and capture the line count of the rules files....

/usr/bin/pulledpork.pl -S 2.9.5.3 -c /etc/pulledpork/pulledpork.conf -k -T -P > 
/dev/null 2>&1 && wc -l /tmp/snortrules/*.rules | grep total
   37727 total
/usr/bin/pulledpork.pl -S 2.9.5.3 -c /etc/pulledpork/pulledpork.conf -k -T -P > 
/dev/null 2>&1 && wc -l /tmp/snortrules/*.rules | grep total
   74659 total
/usr/bin/pulledpork.pl -S 2.9.5.3 -c /etc/pulledpork/pulledpork.conf -k -T -P > 
/dev/null 2>&1 && wc -l /tmp/snortrules/*.rules | grep total
   111603 total

Manually inspecting a file I can see that each run results in each signature 
being duplicated at each update.

What version of the product are you using? On what operating system?

Pulled Pork 0.7.0 on RHEL 6.4

Please provide any additional information below.

This is new behavior with 0.7.0. I was using the same configuration with 0.6.2 
without the duplication.

Original issue reported on code.google.com by scott.p...@gmail.com on 13 Sep 2013 at 3:24

[GoogleCodeExporter]

It appears as though the array of sids within the categories hash is non-unique 
when the '-k' option is used. I can't quite figure out why it's non-unique in 
the first place, but the attached patch takes care of it on the output end.

Original comment by scott.p...@gmail.com on 13 Sep 2013 at 8:54

Attachments:

• pulledpork_0.7.0-unique_sids.patch

[GoogleCodeExporter]

I am seeing Duplicate rules in snort.rules single file output will this patch 
fix that as well?

I have attached part of my logs which are created during snort startup.

Original comment by kingofne...@gmail.com on 12 Mar 2014 at 4:06

Attachments:

• pulledPork0.7.0DuplicateRules.txt

[GoogleCodeExporter]

Confirming that the patch in comment #1 does seem to fix the duplicate sid 
issue for me.  I only checked the VRT rules, didn't check the ET rules.

# This file had 3 days of duplication, so 4 copies of each rule
[root@snort rules]# wc -l VRT-SO-exploit.rules
226 VRT-SO-exploit.rules
... run pulledpork.pl -P -c /etc/pulledpork/pulledpork.conf -v -n -k ...
[root@snort rules]# wc -l VRT-SO-exploit.rules
61 VRT-SO-exploit.rules

Original comment by mrballcb on 24 Mar 2014 at 1:51

[GoogleCodeExporter]

I still get duplicate rules with patch applied. Pulled pork is being run with:
pulledpork.pl -c pulledpork.conf -T

The same rules are in snortrules-snapshot-2956.tar.gz and 
community-rules.tar.gz. They are not deduplicated by pulled pork.

Original comment by tinklusa...@gmail.com on 13 May 2014 at 1:48

[GoogleCodeExporter]

Same issue in my usecase. If i delete all rules in the /etc/snort/rules folder 
and do
su - pulledpork -c "$PULLED_PORK_BIN -c $PULLED_PORK_CONF -k -l to get all VRT 
rules the first time, everything works fine, but if i do it again the rules are 
double. For the third time all rules are triple and so on. This will result in 
a huge output while parsing the config with snort -T -c $CONFIG.

Original comment by daniel.h...@googlemail.com on 14 Oct 2014 at 6:29

[GoogleCodeExporter]

I'm using the default merged /etc/snort/pulledpork.rules

I'm getting duplications from Community rules being merged with, and not 
overwritten by , the paid VRT ruleset.

Jason "the Snake Roberts" Rochon

Original comment by jcroc...@uic.edu on 30 Oct 2014 at 8:36