onsiteURL (and possibly other *URL regexes) breaks on hyphens, underscores

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start an instance of AntiSamy with any policy file.
2. Filter the following tags:

<img src="/whoa_dude" />
<img src="/smiley-frown" />

What is the expected output? What do you see instead?

Expect both tags back and no errors to show. Instead it fails on the value
with the hyphen. Get back the HTML:

<img src="/whoa_dude">

And an error:
The <b>img</b> tag contained an attribute that we couldn't process. The
<b>src</b> attribute had a value of <u>&#47;smiley&#45;frown</u>. This
value could not be accepted for security reasons. We have chosen to remove
the <b>img</b> tag and its contents in order to process this input. 

If we then change "onsiteURL" to the following regex w/ properly escaped
hyphen:

value="([\p{L}\p{N}\\/\.\?=\#&amp;;\-~]+|\#(\w)+)"

Then it trips on the underscore. Resulting HTML:

<img src="/smiley-frown">

Error:
The <b>img</b> tag contained an attribute that we couldn't process. The
<b>src</b> attribute had a value of <u>&#47;whoa&#95;dude</u>. This value
could not be accepted for security reasons. We have chosen to remove the
<b>img</b> tag and its contents in order to process this input. 

So then we add the underscore to the regex to finally get:

value="([\p{L}\p{N}\\/\.\?=\#&amp;;\-_~]+|\#(\w)+)"

And this yields proper results for our narrow little test. Whether other
stuff gets broken or not, I've not checked (pertinent, given how fixing the
hyphen broke the underscore).

What version of the product are you using? On what operating system?

AntiSamy 1.1.1

Please provide any additional information below.

Note that the <img> tag comes out without closing itself for some reason.
This is a separate bug.

I've not tried this with offsiteURL, cssOnsiteUri, or cssOffsiteUri

Original issue reported on code.google.com by thedownw...@gmail.com on 23 May 2008 at 10:26

[GoogleCodeExporter]

I put in the fixes and the two individual <img> tests you show now work, but my
combined tests when combining those two things into the cssOnsiteUri are 
causing me a
headache:

Input: <style> a.test { background-image: url(/smiley-from_dude.gif); } </style>
Output: <style> a.test { } </style>

The <img> tag does not need to be balanced in HTML. Setting the policy 
directive to
use XHTML will cause the <img> tag to be balanced.

Thanks!

Original comment by arshan.d...@gmail.com on 25 May 2008 at 2:02

• Changed state: Started

[GoogleCodeExporter]

Fixed as of June 7th 2008 in SVN. Also added testRegression() test case 
(07/2008) to
ensure URL works in future versions.

Original comment by li.jaso...@gmail.com on 11 Jul 2008 at 11:46

• Changed state: Fixed

[GoogleCodeExporter]

Original comment by arshan.d...@gmail.com on 3 Aug 2009 at 2:38

• Changed state: Verified