DefaultEncoder.canonicalize() Bug

[GoogleCodeExporter]

What steps will reproduce the problem?

1.  Attempt to validate the following HTTPQueryString:

connectid=68470072-44c2-417b-822b-d945dc0364f4&request=GetFeature&service=wfs&ve
rsion=1.1.0&typeName=DigitalGlobe%3AFinishedFeature&bbox=37.5%2C41.5%2C37.8%2C41
.7&PROPERTYNAME=source%2CsourceUnit%2CproductType

2.  Notice the following exception get logged:

2012-08-30 00:02:25,918 WARN  [388e3f54-b1be-442c-a124-45f9fcda7c67] 
[http-8082-10] [IntrusionDetector] - [SECURITY FAILURE Anonymous:null@unknown 
-> /OnlinePortfolio/IntrusionDetector] Invalid input: 
context=com.digitalglobe.dgp.wfs.controller.WfsController, 
type(HTTPQueryString)=^[a-zA-Z0-9()\-=\*\.\?;,+\/:&_ %#-=]*$, 
input=connectid=68470072-44c2-417b-822b-d945dc0364f4&request=GetFeature&service=
wfs&version=1.1.0&typeName=DigitalGlobe:FinishedFeature&bbox=37.5,41.5,37.8,41.7
∝ERTYNAME=source,sourceUnit,productType, 
orig=connectid=68470072-44c2-417b-822b-d945dc0364f4&request=GetFeature&service=w
fs&version=1.1.0&typeName=DigitalGlobe%3AFinishedFeature&bbox=37.5%2C41.5%2C37.8
%2C41.7&PROPERTYNAME=source%2CsourceUnit%2CproductType
org.owasp.esapi.errors.ValidationException: 
com.digitalglobe.dgp.wfs.controller.WfsController: Invalid input. Please 
conform to regex ^[a-zA-Z0-9()\-=\*\.\?;,+\/:&_ %#-=]*$ with a maximum length 
of 1024
        at org.owasp.esapi.reference.validation.StringValidationRule.checkWhitelist(StringValidationRule.java:144)
        at org.owasp.esapi.reference.validation.StringValidationRule.getValid(StringValidationRule.java:306)
        at org.owasp.esapi.reference.DefaultValidator.getValidInput(DefaultValidator.java:213)
        at org.owasp.esapi.reference.DefaultValidator.getValidInput(DefaultValidator.java:185)

3.  Notice that &PROP has been converted to ∝ by the canonicalize method.

4.  If you repeat but put the PROPERTYNAME parameter at the beginning of the 
query string (i.e. without the &), it works fine.

What is the expected output? What do you see instead?

The expected output is that the &PROP is not converted.

What version of the product are you using? On what operating system?

2.0GA

Original issue reported on code.google.com by victropo...@gmail.com on 30 Aug 2012 at 8:03

[GoogleCodeExporter]

As a temporary work around, I've added code to move the PROPERTYNAME parameter 
to the beginning of the query string, if present, but this won't work if there 
are multiple parameters that start with PROP.

Original comment by victropo...@gmail.com on 31 Aug 2012 at 3:08

[GoogleCodeExporter]

Same problem with &or in a string. This is interpreted as logical or and leads 
to a double encoding exception.

I would expect that only ∨ (with the semicolon) is interpreted as a 
HTML-Entity.

Original comment by michael....@freiheit.com on 12 Sep 2014 at 7:48

[GoogleCodeExporter]

Step to reproduce
DefaultEncoder.canonicalize("&OrderBy", false, false)
cononicalizes "&Or" which is causing it to fail validation against 
Validator.HTTPQueryString.

Expected Results : As there is no semi colon next to &or it should not 
canonicalize based on "or"

What version of the product are you using? 
esapi-2.1.0.jar
On what operating system? 
Windows 7

Original comment by sids0...@gmail.com on 29 Oct 2014 at 4:06

[GoogleCodeExporter]

What steps will reproduce the problem?
--------------------------------------
1. Set failstring to: hl=en&ned=us&q=%20foo%20
2. Attempt ESAPI.validator.getValidInput("blah", failString, 
"HTTPParameterValue", 20971520, true, true)
3. Profit...

Traced the issue through the code:

- HTMLEntityCodec.decode takes input string, loops through characters, finds &, 
and calls decodeCharacter(pbs)
- HTMLEntityCodec.decodeCharacter finds "n" as next character, calls 
getNamedEntity(input)
- HTMLEntityCodec.getNamedEntity finds possible to be: "ned=us&q", finds entry 
"ne" (not equals sign) resets input (good), skips the &, skips two chars (ne). 
Does an input.peek for a trailing semicolon. Does not find one, and returns 
entry.getValue().

Thus anything with &<varname> in data that matches a valid entity encoded name 
(without the semicolon) would match an entity encoding even if it's not 
actually an entity encoding, and thus trigger the "mixed codec" rule 
incorrectly. 

There's a chance that it was intentional, but from what I know of entities 
they're supposed to have semicolons on the end. I'm not sure what the intention 
was since the comment "check for a trailing semicolen" wasn't actually useful 
to convey why, just that the code was checking for one. 

Technically this _could_ be used to craft a custom string comprised of nothing 
but entity codec entries if a person could feed data into the validator, as it 
actually converts these non entities into their character counterparts. I could 
see that being bad. 

Here's a snippet of code showing the problem:

    @Test
    public void testGetCanonicalizedStringFail() {
        String failString = "hl=en&ned=us&q=%22foo%22";

        try 
        {
            String result = ESAPI.validator().getValidInput("HTTP Parameter Value", failString, "HTTPParameterValue", 20971520, true, true);

            System.out.println("");
            System.out.println("");
            System.out.println("");
            System.out.println(failString);
            System.out.println("-----------------------------------------------------------------------------------");
            System.out.println("OK: " + result);
        }
        catch (Exception e)
        {
            System.err.println("Caught exception " + e.getClass().getName() + " " + e.getMessage());
            e.printStackTrace();

        }
    }

You can validate the issue using &ted instead of &ned in the string above.

What is the expected output? What do you see instead?
-----------------------------------------------------
The expected output is a properly canonicalized string:
hl=en&ned=us&q="foo"

The output I receive instead is:

SEVERE: [SECURITY FAILURE Anonymous:null@unknown -> 
/ExampleApplication/IntrusionException] INTRUSION - Mixed encoding (2x) 
detected in https://www.foo.com/search?hl=en&ned=us&q=%22foo%22
Caught exception org.owasp.esapi.errors.IntrusionException Input validation 
failure
org.owasp.esapi.errors.IntrusionException: Input validation failure
    at org.owasp.esapi.reference.DefaultEncoder.canonicalize(DefaultEncoder.java:188)
    at org.owasp.esapi.reference.DefaultEncoder.canonicalize(DefaultEncoder.java:122)
    at org.owasp.esapi.reference.validation.StringValidationRule.getValid(StringValidationRule.java:290)
    at org.owasp.esapi.reference.DefaultValidator.getValidInput(DefaultValidator.java:214)
    at com.blah.RequestValidatorExplore.testGetCanonicalizedStringFail(RequestValidatorExplore.java:30)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

What version of the product are you using? On what operating system?
--------------------------------------------------------------------
2.0.1, java happens on any OS using JAVA ESAPI. We grab it from maven using ivy.

Does this issue affect only a specified browser or set of browsers?
-------------------------------------------------------------------
Not browser specific.

Please provide any additional information below.
------------------------------------------------
I assume the fix is:

--- HTMLEntityCodec.java    2015-01-13 15:56:38.807077692 -0400
+++ HTMLEntityCodecNew.java 2015-01-13 15:58:06.245605842 -0400
@@ -267,6 +267,8 @@
        // check for a trailing semicolen
        if(input.peek(';'))
            input.next();
+       else
+           return null;

        return entry.getValue();
    }

Original comment by happyha...@gmail.com on 14 Jan 2015 at 2:09