Encoding of = in querystring

[hlidotbe]

Hi,

I have a strange bug with query strings.

This is my http file:

GET https://[redacted/path/to/something/?category=tests
Accept-Language: fr

And if I run it, the = gets encoded as %3d and the request fails:

#+RES
array(1) {
  ["category=tests"]=>
  string(0) ""
}

#+END

Running with curl sends the request correctly and the array is populated as expected.

[kevintraver]

Can confirm this is not working:

GET http://httpbin.org/get?foo=bar

returns:

{
  "args": {
    "foo=bar": ""
  },
  "url": "http://httpbin.org/get?foo%3dbar"
}

[NTBBloodbath]

Hey, this is because we encode the request query parameters by using cURL with the equivalent of the curl command-line utility --data-urlencode.

And this definitely is a problem, perhaps we could manually remove the encoding for = again after cURL is done with encoding the parameters?

[NTBBloodbath]

Note to self: it might be a good idea to take a look at this stackoverflow post.

[hlidotbe]

Hey, this is because we encode the request query parameters by using cURL with the equivalent of the curl command-line utility --data-urlencode.

And this definitely is a problem, perhaps we could manually remove the encoding for = again after cURL is done with encoding the parameters?

That might break things differently. What if I have a fully encoded url already ? As it is I guess you will double encode it. But if I need a = symbol in some value then you will decode it back ? Something like ?key=a%3db could be double encoded then the = replaced which should work but then I can't have any other percent encoded value. ?key=a%3db&key2=value%20with%20spaces would be sent as ?key=a%3db&key2=value%2520with%2520spaces

I'd rather be responsible of the correct encoding of the querystring than having to guess what's going on under the hood.

[NTBBloodbath]

Hey, this is because we encode the request query parameters by using cURL with the equivalent of the curl command-line utility --data-urlencode. And this definitely is a problem, perhaps we could manually remove the encoding for = again after cURL is done with encoding the parameters?

That might break things differently. What if I have a fully encoded url already ? As it is I guess you will double encode it. But if I need a = symbol in some value then you will decode it back ? Something like ?key=a%3db could be double encoded then the = replaced which should work but then I can't have any other percent encoded value. ?key=a%3db&key2=value%20with%20spaces would be sent as ?key=a%3db&key2=value%2520with%2520spaces

I'd rather be responsible of the correct encoding of the querystring than having to guess what's going on under the hood.

Hmm you got a point here. It is an edge case but it's still possible for it to happen, what would you recommend doing?

You can temporarily disable the URL encoding with the encode_url configuration option until we find a good solution, by the way.

[flexagoon]

@NTBBloodbath I think the best option would be to split the query string and encode the values individually, then add them to the curl url by using the CURLU_APPENDQUERY flag. This is what the curl command line --data-urlencode option does: it splits the string by = and only encodes the second part.

I'll probably try to submit a PR that does this.