search

curlconverter / curlconverter

Transpile curl commands into Python, JavaScript and 27 other languages
https://curlconverter.com
MIT License
7.54k stars 934 forks source link

Parse lowercase cookie headers #89

Closed Informatic closed 2 years ago

Informatic commented 6 years ago

Sometimes all headers in curl line seem to be lowercased (eg. requests to/from humblebundle.com) and cookies are only parsed from "Cookie" headers.

NickCarneiro commented 6 years ago

Can you paste an example?

Informatic commented 6 years ago

Something like that: (URL accesible after logging in, but that doesn't really matter, browser version in user-agent)

curl 'https://www.humblebundle.com/home/library' -H 'pragma: no-cache' -H 'cookie: csrf_cookie=...; hbguard="..."; pnctest=1; session-set=true; utmcsr="Humble Bundle Newsletter"; utmccmpn=...; utmcmed=email; crdl_humblebundlelivecID=...; amazon-pay-abtesting-new-widgets=false; _simpleauth_sess="..."' -H 'dnt: 1' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'cache-control: no-cache' -H 'authority: www.humblebundle.com' -H 'referer: https://www.humblebundle.com/' --compressed
Informatic commented 6 years ago

Ah, interesting. This actually seems like a HTTP/2 (h2?) request, so that's why it's all lowercase.

NickCarneiro commented 6 years ago

This has opened a whole new can of worms.

On my macbook with curl 7.54.0 it looks like http2 is used by default for sites that support it.

curl -v https://www.google.com < HTTP/2 200

But it looks like the current version of python requests does not support it, and furthermore, the upcoming version MIGHT support it: https://www.kennethreitz.org/requests3

NickCarneiro commented 6 years ago

I think the best option for now is to add some kind of caveats page that explains the state of http2 or detects all lowercase headers and adds a warning about http2 in a comment in the generated code.

Informatic commented 6 years ago

I think it would be pretty useful to still parse cookies properly (by just detecting "Cookie:" header case-insensitively here and there), though, as most HTTP2 services support HTTP/1.1 as well.

NickCarneiro commented 6 years ago

You're right. I'll fix this soon.