%2F if specified in path is getting double encoded as %252F iresspective of setting RestAssured.urlEncodingEnabled=false

[GoogleCodeExporter]

What steps will reproduce the problem?

 RestAssured.urlEncodingEnabled=false
given().pathParam("pageUrl", 
"Product%2F1756%2FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs").
    get("/findItem;pageUrl={pageUrl}")

 OR

get("/findItem;pageUrl=Product%2F1756%2FMilk-Bone-Flavor-Snacks-Biscuits-for-Dog
s;sortby=true;test=abc123")

The server is receiving the pageURl as 
"Product%252F1756%252FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs" .
you can clearly see %2F is getting double encoded as %252F.

if you use any of the above methods you can see when the pageUrl parameter 
passed to the server is double encoded. There is no way to pass a matrixparam 
to a rest service which contains "/" . if possible add support for matrixparam 
as well, so that  we will be able to test rest services which takes multiple 
matrixparam as input. 

What is the expected output? What do you see instead?

i should have seen 
pageUrl=Product%2F1756%2FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs at the 
server. Bu i see its coming as 
Product%252F1756%252FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs

What version of the product are you using? On what operating system?

1.6.2 on Windows 7

Please provide any additional information below.

Original issue reported on code.google.com by alok.me...@julysystems.com on 27 Jun 2012 at 7:20

[GoogleCodeExporter]

Was already reported: http://code.google.com/p/rest-assured/issues/detail?id=175

However, that ticket has status "Fixed" although the problem was only partly 
resolved.

Original comment by mi.koel...@googlemail.com on 28 Jun 2012 at 2:06

[GoogleCodeExporter]

The issue still persist . Any workarounds ?

Original comment by alok.me...@julysystems.com on 29 Jun 2012 at 7:50

[GoogleCodeExporter]

You can try if using 

given().urlEncodingEnabled(false).pathParam(...)

works.

Original comment by johan.ha...@gmail.com on 16 Aug 2012 at 10:57

[GoogleCodeExporter]

No, in 1.6.2 that does not help.

Original comment by mi.koel...@googlemail.com on 3 Sep 2012 at 9:26

[GoogleCodeExporter]

Ok thanks for trying.

Original comment by johan.ha...@gmail.com on 3 Sep 2012 at 9:28

• Changed state: Accepted

[GoogleCodeExporter]

I tried with both 1.6.1 and 1.6.2

When I turn urlEscapeEnabled(false) RA allows me to pass query string values 
that are already escaped in and the items are NOT double escaped:

given().urlEncodingEnabled(false)
             .get("/rest/api/2.0.alpha1/search?foo%20=%20BAM%20AND%20issuetype%20=%20Bug")

GET /rest/api/2.0.alpha1/search?foo%20=%20BAM%20AND%20issuetype%20=%20Bug 
HTTP/1.1

However if the escaping occurs in the actual path, double escaping occurs

given().urlEncodingEnabled(false)
             .get("/rest/api/2.0.alpha1/search/foo%20=%20BAM%20AND%20issuetype%20=%20Bug")

GET 
/rest/api/2.0.alpha1/search/foo%2520=%2520BAM%2520AND%2520issuetype%2520=%2520Bu
g HTTP/1.1

I have some fancy URIs that I want to test with, but can't figure out how to 
NOT get double escaping (thus failing my tests).  For instance I have some URI 
path segments that have the ?

I believe 188 is a subset of this problem.  Either I need to escape the full 
path myself (urlEncodingEnabled(false) working) or some more characters need to 
be better encoded.  Or both. Below is my list:

These get encoded correctly

"dot .", "EURO \u20ac", "PARA \u00a7", "colon :",
"asterisk *", "parens ()", "curly {}", "angle <>", "square []", "percent %", 
"and &",
"comma ,", "hash #", "plus +", "tilde ~"

and the ones that are not escaped correctly 

"qmark ?" //bug 188
"backslash \\", 
"slash /", 
"semicolon ;", 
"quotes \"'", 
"surrogate pair \ud800\udc00"

Any quick workarounds?

Original comment by kwig...@gmail.com on 7 Sep 2012 at 4:13

[GoogleCodeExporter]

also got bitten by this issue (1.6.2 also).
hopefully, the resource I was testing was just a GET resource, so I used 
java.net.URL and hamcrest matchers as a workaround:
    URL url = new URL(RestAssured.baseURI + ":" + RestAssured.port + "/api/agents/probeUrl/" + URLEncoder.encode("funky://characters&to&*escape", "UTF-8"));
    InputStream openStream = url.openStream();
    String result = new java.util.Scanner(openStream).useDelimiter("\\A").next();
    String expectedContent = "{\"agentId\":\"embedded\",\"available\":true";
    assertThat(result, containsString(expectedContent));

Original comment by anthony....@gmail.com on 20 Sep 2012 at 4:03

[GoogleCodeExporter]

Original comment by johan.ha...@gmail.com on 11 Oct 2012 at 7:06

• Changed state: Fixed

[GoogleCodeExporter]

Can you describe the work around? 

When I try to send this URL with the code shown below I get a 404 because the 
%2F, is rest assured decoding the %2F somewhere?

http://localhost:8097/something/ceec6bdd-3734-41f3-b13d-ab722a45eee6/QumrSExq2yL
cuISsau2%2Fj0BIdwuQATk4wUCCTBHzO5k%3D/integrationTestGame/WIzIgbLkEEgMp9cixKmC23
FvqzYOhEVUFnmzpkWC3jo%3D

given().urlEncodingEnabled(false).get(url).asString();

Original comment by 0xffff...@gmail.com on 25 Oct 2012 at 9:19

[GoogleCodeExporter]

Does it work if you're not using rest assured?

Original comment by johan.ha...@gmail.com on 4 Nov 2012 at 4:57

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Looks like the original reporter dropped out, but I can vouch that we're still 
seeing this bug in 1.8.1. I ran some tests and checked what the server ended up 
receiving in each case.

String val = "yes/no";
String encoded = URLEncoder.encode(val, "UTF-8");

// Case 1: val is encoded
given().queryParam("val", val).get("https://example.com/foo"); // 
/foo?val=yes%2Fno

// Case 2: val is not encoded
given().urlEncodingEnabled(false).queryParam("val", 
val).get("https://example.com/foo"); // /foo?val=yes/no

// Case 3: val is encoded
given().pathParam("val", val).get("https://example.com/foo?val={val}"); // 
/foo?val=yes%2Fno

// Case 4: val is not encoded
given().pathParam("val", val).get("https://example.com/foo/{val}/something"); 
// /foo/yes/no/something

// Case 5: "encoded" is encoded (so now it's double-encoded)
given().pathParam("encoded", 
encoded).get("https://example.com/foo/{encoded}/something"); // 
/foo/yes%252Fno/something

// Case 6: val is not encoded
given().urlEncodingEnabled(false).pathParam("val", 
val).get("https://example.com/foo/{val}/something"); // /foo/yes/no/something

// Case 7: encoded has been un-encoded
given().urlEncodingEnabled(false).pathParam("encoded", 
encoded).get("https://example.com/foo/{encoded}/something"); // 
/foo/yes/no/something

Cases 1-3 work as I would expect.

Case 4 seems inconsistent with case 3, but maybe that's intentional. However, 
if case 4 is correct, then I would not expect the double-encoding in case 5.

Case 6 seems correct, but then case 7 is baffling; why does the value get 
*un*-encoded?

In the end, I seem to be able to send requests that are either double-encoded 
or not encoded at all, but never just properly encoded.

Original comment by John.Pr...@gmail.com on 30 Oct 2013 at 1:59

[GoogleCodeExporter]

Thanks for your thorough investigation. I've broken my hand and have a really 
hard time to write and code right now so please help out!

Original comment by johan.ha...@gmail.com on 30 Oct 2013 at 2:02

• Changed state: Accepted

[GoogleCodeExporter]

I'm also experiencing John's Case 7; my "%2F"s are getting decoded to "/".

Original comment by gavinhac...@gmail.com on 31 Oct 2013 at 5:28

[GoogleCodeExporter]

I poked through the code yesterday but couldn't find anything obvious. 
Unfortunately, I was unable to get Groovy debugging working well enough to 
really track things down and eventually had to move on to other work. If I get 
a chance to dig further I'll post any findings, or maybe the test cases will be 
of use to someone else.

Original comment by John.Pr...@gmail.com on 31 Oct 2013 at 5:48

[GoogleCodeExporter]

I don't think it's valid to have a "/" in the "resource path" (e.g. yes/no 
before the ?) before the query parameters even though you encode it. From my 
understanding severs decode everything specified after the host (e.g. 
https:://example/foo). So if "yes/no" would be properly URL encoded by REST 
Assured it still wouldn't work on the server side.

Original comment by johan.ha...@gmail.com on 15 Nov 2013 at 9:24

[GoogleCodeExporter]

What I mean by "wouldn't work on the server side" is that if the server expects 
a path parameter called "yes/no".

Original comment by johan.ha...@gmail.com on 15 Nov 2013 at 9:25

[GoogleCodeExporter]

We're using Jersey on the server side and it *does* work if things are properly 
encoded by the client. Eg. given

@Path("/foo/{val}/something")
public void doGet(@PathParam("val") String val)

the value of "val" will be "foo/bar" if the slash was encoded as %2F by the 
client.

I did a quick search and it appears that some servers disable support for this 
(eg. Apache: 
http://stackoverflow.com/questions/3235219/urlencoded-forward-slash-is-breaking-
url), but it's not strictly "wrong" as far as I can tell.

That said, in the most recent version of our API we've eliminated the need to 
ever do this, but we still need to maintain tests for the old version until we 
can actually remove support for it.

Original comment by John.Pr...@gmail.com on 15 Nov 2013 at 3:36

[GoogleCodeExporter]

Thanks for sharing. I'm working on a fix for this, lots of changes though but 
hopefully I'll be able to get it right sometime next week.

Original comment by johan.ha...@gmail.com on 15 Nov 2013 at 5:06

[GoogleCodeExporter]

I've implemented a new snapshot release in which I've tried to fix all URL 
encoding issues. Please try it out by depending on version 1.8.2-SNAPSHOT after 
having added the following maven repo:

<repositories>
        <repository>
            <id>sonatype</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
            <snapshots />
        </repository>
</repositories>

It would be great if you can try out asap since I really want to make a new 
release.

Original comment by johan.ha...@gmail.com on 18 Nov 2013 at 2:41

[GoogleCodeExporter]

hello Johan,
I run several tests with 1.8.2-SNAPSHOT, nothing changed for me :
https://gist.github.com/anthonydahanne/7531730

Original comment by anthony....@gmail.com on 18 Nov 2013 at 5:25

[GoogleCodeExporter]

Thanks a lot for trying it out so soon. There seem to be several (new) issues 
that you've discovered that my tests doesn't find. I'll try to look into them 
soon.

Original comment by johan.ha...@gmail.com on 18 Nov 2013 at 8:52

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

@Anthony: I've looked into the code that you provided and these are my findings:

  1. I believe this to be correct since the URL is more or less invalid if you disable URL encoding in this case?
  2. This is probably correct because there's no way for REST Assured to know how to URL encode the path: "http://localhost/agents/probeUrl/https://localhost:9888 How should RA know that it only should url encode stuff after "http://localhost/agents/probeUrl/ ? If you use named or unnamed path parameters then it'll work, for example given().pathParam("x", "https://localhost:9888").when().get("/agents/probeUrl/{x}"); .
  3. I cannot reproduce this. I've checked both the REST Assured logs and tried caught the request with Wireshark and RA is not doing double URL encoding as far as I can tell.
  4. Again I've checked with Wireshark and RA does indeed send the request (and does double URL encoding). I've added test cases as well (see the last four tests in URLEncodingITest)

Please let me know if you still think there's something wrong.

Original comment by johan.ha...@gmail.com on 19 Nov 2013 at 12:45

[GoogleCodeExporter]

Actually regarding (1) I think that it ought to be possible to specify URL's 
with "://" in the path (besides the scheme). I'll fix this but it's actually 
another issue.

Original comment by johan.ha...@gmail.com on 19 Nov 2013 at 12:57

[GoogleCodeExporter]

@Johan,
thanks a lot for investigating.
1) yeah this is expected behavior
2) I did not try with .pathParam() ; I just tested it, it works (I updated the 
gist at https://gist.github.com/anthonydahanne/7531730
3) that is weird, if I have time I'll create a standalone maven project to 
reproduce the issue
4) ok, that sounds expected anyway.

all in all, I like the pathParam solution; as you said, it makes sense to 
extract the part of the url that needs encoding into a path param, so that RA 
knows what needs to be encoded. 
thanks a lot for your help, now I can get rid of HttpClient in my tests :-)

Original comment by anthony....@gmail.com on 19 Nov 2013 at 2:53

[GoogleCodeExporter]

You can also use unnamed path parameters like this: .. 
.get("/agents/probeUrl/{x}", "https://localhost:9888");

I'll consider this issue fixed now. If anyone find any further issues please 
let me know asap.

Thanks for your patience and help!

Original comment by johan.ha...@gmail.com on 19 Nov 2013 at 5:59

• Changed state: Fixed