search

spulec / uncurl

A library to convert curl requests to python-requests.
Apache License 2.0
607 stars 94 forks source link

uncurl.parse response puts headers in alphabetical order #37

Open machine-gurning opened 4 years ago

machine-gurning commented 4 years ago

G'day there, I am using this python package in the hope that it does something identical to curl.trillworks.com.

Mostly works hunky-dory, but I note that the website mentioned above returns a headers dictionary in the correct headers order, whereas the uncurl.parse() function gives me an unordered dictionary, in alphabetical order. It it possible to change this to an ordered dictionary?

Input:

uncurl.parse("curl 'https://www.amazon.com/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1'")

Output: 'requests.get("https://www.amazon.com/",\n headers={\n "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",\n "Accept-Language": "en-US,en;q=0.5",\n "Connection": "keep-alive",\n "DNT": "1",\n "Upgrade-Insecure-Requests": "1",\n "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0"\n },\n cookies={},\n)'

Desired output:

headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.5', 'DNT': '1', 'Connection': 'keep-alive', 'Upgrade-Insecure-Requests': '1', }

Cheers and thanks for your work

spulec commented 4 years ago

Hmmm, that is a fair point. We are being inaccurate by sorting headers.

It is really nice to have them sorted from a visual perspective though :/

Would we still want to sort cookies too or no?

machine-gurning commented 4 years ago

May I suggest an option to either return the output in alphabetical order for easy comprehension, or unsorted order (i.e. the order of curl.trillworks.com, and the order of the headers generated automatically by browsers) for the purpose of accuracy?

Such an option I don't think has been offered before, as I have been fishing around on forums for some time now trying to find it

spulec commented 4 years ago

Hey, can you take a look at this PR and see if it matches what you are thinking: https://github.com/spulec/uncurl/pull/38