Loading XML file './default_filter.xml'... The rule '(?:union....cannot be compiled properly

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Downloaded default xml from 
https://dev.itratos.de/projects/php-ids/repository/raw/trunk/lib/IDS/default_fil
ter.xml
2.
./scalp.py -l /var/log/apache2/access.log -f ./default_filter.xml -o file --html
3.

What is the expected output? What do you see instead?
something...ERROR: "(XML)...cannot be compiled properly"

What version of the product are you using? On what operating system?
0.4

Please provide any additional information below.

Original issue reported on code.google.com by jnahan...@gmail.com on 20 Aug 2011 at 7:37

[GoogleCodeExporter]

I have the same isue with scalp-0.4.py on SUSE Linux Enterprise Server 11 
(x86_64)2.6.32.12-0.7-default.
Output:
The rule 
'(?:union\s*(?:all|distinct|[(!@]*)?\s*[([]*\s*select)|(?:\w+\s+like\s+\")|(?:li
ke\s*"\%)|(?:"\s*like\W*["\d])|(?:"\s*(?:n?and|x?or|not 
|\|\||\&\&)\s+[\s\w]+=\s*\w+\s*having)|(?:"\s*\*\s*\w+\W+")|(?:"\s*[^?\w\s=.,;)(
]+\s*[(@"]*\s*\w+\W+\w)|(?:select\s*[\[\]()\s\w\.,"-]+from)|(?:find_in_set\s*\()
' cannot be compiled properly

Original comment by yts...@gmail.com on 10 Oct 2011 at 12:13

[GoogleCodeExporter]

Same issue : 
The rule 
'(?:union\s*(?:all|distinct|[(!@]*)?\s*[([]*\s*select)|(?:\w+\s+like\s+\")|(?:li
ke\s*"\%)|(?:"\s*like\W*["\d])|(?:"\s*(?:n?and|x?or|not 
|\|\||\&\&)\s+[\s\w]+=\s*\w+\s*having)|(?:"\s*\*\s*\w+\W+")|(?:"\s*[^?\w\s=.,;)(
]+\s*[(@"]*\s*\w+\W+\w)|(?:select\s*[\[\]()\s\w\.,"-]+from)|(?:find_in_set\s*\()
' cannot be compiled properly

Operating system :
Debian Linux 5.0.8 with Python 2.5.2 and scalp 0.4

Log format used for my access logs : 
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"

Original comment by kevin.le...@reezorp.com on 26 Oct 2011 at 10:15

[GoogleCodeExporter]

Same problem:

Fedora Core 15

python-2.7.1-7.fc15.x86_64

Original comment by yje...@security-projects.com on 12 Nov 2011 at 6:45

[GoogleCodeExporter]

any news about? I have the sape problem 

Red Hat Enterprise Linux ES release 4 (Nahant Update 6)
python-2.3.4-14.4

when trying to exclude the rules, another rule gives the error...

thanks

Original comment by manu.ita...@gmail.com on 3 Jan 2012 at 4:50

[GoogleCodeExporter]

$ ./scalp-0.4.py --log all.parsed --attack xss,sqli,lfi,ref
Loading XML file 'default_filter.xml'...
The rule 
'(?:union\s*(?:all|distinct|[(!@]*)?\s*[([]*\s*select)|(?:\w+\s+like\s+\")|(?:li
ke\s*"\%)|(?:"\s*like\W*["\d])|(?:"\s*(?:n?and|x?or|not 
|\|\||\&\&)\s+[\s\w]+=\s*\w+\s*having)|(?:"\s*\*\s*\w+\W+")|(?:"\s*[^?\w\s=.,;)(
]+\s*[(@"]*\s*\w+\W+\w)|(?:select\s*[\[\]()\s\w\.,"-]+from)|(?:find_in_set\s*\()
' cannot be compiled properly

$ python --version
Python 2.7.1

Original comment by jca...@gmail.com on 9 Jan 2012 at 8:13

[GoogleCodeExporter]

Same here.
Python 2.5.2

Original comment by night.k...@gmail.com on 9 Jan 2012 at 11:23

[GoogleCodeExporter]

Ok, it seems that there is an extra \ in (?:\w+\s+like\s+\") in front of the ".
But removing it leads to:
sre_constants.error: nothing to repeat

it usually appears with nested 0 something quantifiers (like (x?)* or (x*)*), 
but don't see where that case happens.

Original comment by night.k...@gmail.com on 10 Jan 2012 at 12:39

[GoogleCodeExporter]

Ok, found it... (?:all|distinct|[(!@]*)? should be (?:all|distinct|[(!@]+)?

Original comment by night.k...@gmail.com on 10 Jan 2012 at 12:44

[GoogleCodeExporter]

Next: The rule '(?i:(\%SYSTEMROOT\%))' cannot be compiled properly... the 
construct (?i: (case insensitive non-capturing group) is not supported... 
should be uglyfied as
(?:(\%[sS][yY][sS][tT][eE][mM][rR][oO][oO][tT]\%))

And that's it, for me default.xml is loaded after those two changes.

Original comment by night.k...@gmail.com on 10 Jan 2012 at 12:51

[GoogleCodeExporter]

Thanks!  Got mine to run!

Original comment by vince...@gmail.com on 26 Mar 2012 at 4:31

[GoogleCodeExporter]

default_filter.xml with night.k...@gmail.com fixes applied available here: 
http://pastebin.com/uDziqcD5

Original comment by joshuasc...@gmail.com on 30 Apr 2012 at 4:37

[GoogleCodeExporter]

Thank you! Helped me out :)

Original comment by JakeCh...@gmail.com on 21 May 2012 at 8:46

[GoogleCodeExporter]

Thanks, the two fixes mentioned here fixed the issue. Why is the xml file not 
updated on the home page of the scalp project? 

Original comment by sgyan...@gmail.com on 28 May 2012 at 8:48

[GoogleCodeExporter]

Thanks! Mine worked smooth after the fix :)

Original comment by mattlogb...@gmail.com on 1 Aug 2012 at 1:46

[GoogleCodeExporter]

thank you guys :)

Original comment by justice....@gmail.com on 7 Oct 2012 at 10:06

[GoogleCodeExporter]

Thanks for the fix and the pastebin.

Original comment by AndreTim...@gmail.com on 23 Jan 2013 at 3:56

[GoogleCodeExporter]

Worked for me too (Python 2.5.1 on Windows), thanks.

Original comment by a...@reece.myzen.co.uk on 18 Aug 2013 at 9:04

[GoogleCodeExporter]

Yey its working so you have to replace 

(?:all|distinct|[(!@]*)?   with   (?:all|distinct|[(!@]+)?

and 

(?i:(\%SYSTEMROOT\%))     with    
(?:(\%[sS][yY][sS][tT][eE][mM][rR][oO][oO][tT]\%))

It will then run :D have a nice day

Original comment by 7ical...@gmail.com on 21 Mar 2014 at 9:45

[GoogleCodeExporter]

Had the same issue on centos , those fixes mention fixed it.

Original comment by bento...@gmail.com on 19 Aug 2014 at 12:23

[GoogleCodeExporter]

Excelente solution by 7ical...@gmail.com just replace
(?:all|distinct|[(!@]*)?   with   (?:all|distinct|[(!@]+)?
and 
(?i:(\%SYSTEMROOT\%))     with    
(?:(\%[sS][yY][sS][tT][eE][mM][rR][oO][oO][tT]\%))

Thank you.

Original comment by ger....@gmail.com on 14 Oct 2014 at 6:58