Web version does not display header "Content-Type" (curl to python)

[vovavovavovavova]

Converting Curl to python requests 14.03.2023

Missing "Content-Type" header, in python result. I checked this header in curl pasting, it exists. -H 'content-type: application/x-www-form-urlencoded' -H 'user-agent: Mozilla/5.0...

Previously it was ok and header displayed correcttly.

[verhovsky]

It's deleted when it becomes redundant, Requests will add that header with that value when you include a data= argument.

[vovavovavovavova]

it's not redundant, if I emulate request with python without for example urlencoded header, using the output of CurlConverter (where data is shown as a javascript object / python dict), it will be a bit different request!

[verhovsky]

Can you give me an example command?

[vovavovavovavova]

example command curl -H 'Host: xxx' -H 'content-type: application/x-www-form-urlencoded' --data-binary "action=past&type=1" --compressed 'https://mysite/endpoint'

will be converted and displayed at web interface as

import requests

headers = {
    'Host': 'xxx',
}

data = {
    'action': 'past',
    'type': '1',
}

response = requests.post('https://mysite/endpoint', headers=headers, data=data)

The shown output WILL NOT emulate original request.

[verhovsky]

Thank you. We can start a simple server on port 8888 with nc -kl 8888 and send this request with curl to it:

curl -H 'Host: xxx' -H 'content-type: application/x-www-form-urlencoded' --data-binary "action=past&type=1" --compressed 'http://localhost:8888/endpoint'

and see that curl 7.88.1 sends this HTTP:

POST /endpoint HTTP/1.1
Host: xxx
User-Agent: curl/7.88.1
Accept: */*
Accept-Encoding: deflate, gzip, br, zstd
content-type: application/x-www-form-urlencoded
Content-Length: 18

action=past&type=1

If we run the Python code that that command is converted to

import requests

headers = {
    'Host': 'xxx',
}

data = {
    'action': 'past',
    'type': '1',
}

response = requests.post('http://localhost:8888/endpoint', headers=headers, data=data)

we get this HTTP

POST /endpoint HTTP/1.1
User-Agent: python-requests/2.28.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Host: xxx
Content-Length: 18
Content-Type: application/x-www-form-urlencoded

The curl command sends content-type: application/x-www-form-urlencoded and the Python code sends Content-Type: application/x-www-form-urlencoded. Requests adds the Content-Type header here

https://github.com/psf/requests/blob/7f694b79e114c06fac5ec06019cada5a61e5570f/requests/models.py#L567-L569

[verhovsky]

I changed it to only remove the redundant content-type header if it's the only header. https://github.com/curlconverter/curlconverter/commit/7548d4ca47c6d060d43560167053ec77aea65625

Your command is now converted to

import requests

headers = {
    'Host': 'xxx',
    'content-type': 'application/x-www-form-urlencoded',
}

data = {
    'action': 'past',
    'type': '1',
}

response = requests.post('http://localhost:8888/endpoint', headers=headers, data=data)

Is that good?

[vovavovavovavova]

my request had a lot more headers, that was just a stripped example.

But if it is correctly added by requests by default, then it is not an issue.

[verhovsky]

If you have a curl command that is working but the converted Python code isn't, that's a problem and I hope you open another issue on GitHub so that we can fix it.

As I said, you can try changing the URL to localhost:8888, and running a server with nc -kl 8888 then pasting the new command with the localhost:8888 URL into curlconverter and running the command and the code and comparing the two requests to try to figure out why one isn't working.