python-requests formatter when using parameters in x-www-form/urlencoded

[erral]

We are using this extension in a Sphinx based documentation site where we have to show the following HTTP request:

POST /@@oauth2-token HTTP/1.1
Accept: application/json
Content-type: application/x-www-form-urlencoded

grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=REDACTED

This is converted to python-requests formatter like this (note the data parameter):

requests.post('http://nohost/@@oauth2-token', headers={'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'}, data='grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=REDACTED')

But it should be like this (note the data parameter):

requests.post('http://nohost/@@oauth2-token', headers={'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'}, data={'grant_type':'urn:ietf:params:oauth:grant-type:jwt-bearer', 'assertion':'REDACTED'})

At least, that's what it works in our setup, otherwise we are getting errors because the backend doesn't find the grant_type parameter.

[datakurre]

@erral Nice catch. I agree that we need to fix this.

You are first to report this, because the current formatter works until values include characters (here colons) which should be URL encoded.

[datakurre]

@erral Now I have a second thought, would this produce working example:

POST /@@oauth2-token HTTP/1.1
Accept: application/json
Content-type: application/x-www-form-urlencoded

grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=REDACTED

Because, that is the version I see being produced by your request call version.

Yet, I still agree that your version would be more "pythonic" for requests.

[erral]

let me do a test and report back

[datakurre]

Anyway, it's probably safe to do parse_qs for application/x-www-form-urlencoded request data and pass that to requests. Both encoded and non-encoded versions would result the same example in this case:

https://github.com/collective/sphinxcontrib-httpexample/pull/89

[erral]

Using the %3A trick did not work, the HTTP request looks like this:

POST /@@oauth2-token HTTP/1.1
Accept: application/json
Content-type: application/x-www-form-urlencoded

grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=REDACTED

And the python-requests output like this:

requests.post('http://nohost/@@oauth2-token', headers={'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'}, data='grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=REDACTED')

[erral]

With the changes in the PR, it works perfectly.

[datakurre]

Thanks for testing. Yet, I'm still surprised. Before I asked you to test, I tested requests against https://hub.docker.com/r/mendhak/http-https-echo and both

requests.post('http://localhost:8081', headers={'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'}, data={'grant_type':'urn:ietf:params:oauth:grant-type:jwt-bearer', 'assertion':'REDACTED'})

and

requests.post('http://localhost:8081', headers={'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'}, data='grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=REDACTED')

Showed similar HTTP request being sent. (Both had grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=REDACTED as HTTP request body.)

But good to hear that the patched version works. It probably results in better user experience too,

[erral]

I dont't know... perhaps the backend makes some assumption?

I haven't tested the backend sending the encoded colons... I have just tested the Sphinx output

[datakurre]

Thanks. I'll see, if I can fix issues with the latest Sphinx (as in https://github.com/collective/sphinxcontrib-httpexample/issues/86) and will cut a release soonish.