search

python-hyper / hyper

HTTP/2 for Python.
http://hyper.rtfd.org/en/latest/
MIT License
1.05k stars 191 forks source link

HTTP20Adapter sends extra headers ?? #416

Open qwerty32123 opened 5 years ago

qwerty32123 commented 5 years ago 

import hashlib
import requests
import json
from tls_version import MyAdapter
import collections
from userdata import  UserData
import time
from random import randrange
from hyper.contrib import HTTP20Adapter

headers2 = [('Upgrade-Insecure-Requests', '1'),
        ('User-Agent', 'Mozilla/5.0 (Linux; Android 5.1.1; google Pixel 2 Build/LMY47I; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36'),
        ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'),
        ('Accept-Encoding', 'gzip, deflate'),
        ('Accept-Language', 'es-ES,es;q=0.9,en-US;q=0.8,en;q=0.7'),

        ]
class post_calls():
        def start(self,headers_pass,body_pass,params,url,method):
            proxies = {
                'http': ip,
                'https': ip
            }
            body = str(body_pass)

            #send the POST request
            session = requests.session()
            session.mount('https://', MyAdapter())
            session.headers =  collections.OrderedDict(headers_pass)
            if method == 'get':
                q = 'https://' + server + '.' + host
                q = q.replace('.www.', '.')
                session.mount('https://', HTTP20Adapter())
                print('q='+q)
                response = session.get(url, proxies=proxies, params=params, verify=charlesproxy)

def login_world2(sid): 

    a = post_calls()
    q ='https://'+server+'.'+ host+'/login.php?mobile&sid='+sid+'&2'
    q = q.replace('.www.','.')
    params = {}
    url = q
    body = '0'
    login = a.start(headers2,body,params,url,'get')
    return login

if __name__ == "__main__":
    login_get = login_world(sid)
    print(login_get)

and it sends those headers(im sniffing my code http/s traffic with charlesproxy)

:method: GET
:scheme: https
:authority: server.url.com
:path: /login.php?mobile&sid=577f0967545d6acec94716d265dd4867fa4db4a446326ecde7486a97feede14702f4911438f4a4cd097677f0dd962786ef14b3f16f1184ee63a506155d522f53&2
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 5.1.1; google Pixel 2 Build/LMY47I; wv) AppleWebKit/537.36 (KHTML
user-agent: like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36
accept: text/html
accept: application/xhtml+xml
accept: application/xml;q=0.9
accept: image/webp
accept: image/apng
accept: */*;q=0.8
accept: application/signed-exchange;v=b3
accept-encoding: gzip
accept-encoding: deflate
accept-language: es-ES
accept-language: es;q=0.9
accept-language: en-US;q=0.8
accept-language: en;q=0.7

i need to send them with orderectdict like these(with only ) requests goes good problem is thath server is http 2.0 so i need pseudo headers so i need hyper 

:method: GET
:authority: server.url.com
:scheme: https
:path: /login.php?mobile&sid=2ea530a62cb63af6c14be116b7df86ad85cd77c9a11aa3c881b3a460e6c14fbd1fd8b79bd66c9782073705cdff25e890e65b5aeb852fde24c2d54a6e4ee49890&2
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 5.1.1; google Pixel 2 Build/LMY47I; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate
accept-language: es-ES,es;q=0.9,en-US;q=0.8,en;q=0.7
VeNoMouS commented 5 years ago

Yea im seeing something sort of simular wit the accept-range and age

cache-control:           max-age=0, must-revalidate, no-cache, no-store
content-type:            application/json
expires:                 Sat, 28 Jul 2018 10:26:42 GMT
pragma:                  no-cache
set-cookie:              PHPSESSID=45511e9a5f378584c2d90e5494251808; expires=Sun, 28-Jul-2019 11:26:42 GMT; Max-Age=3600; path=/; domain=xxxxxxxx.com; secure; HttpOnly
set-cookie:              X-Magento-Vary=a74fedc5efc1d1867a421578922f820cb12e0565; expires=Sun, 28-Jul-2019 11:26:42 GMT; Max-Age=3600; path=/; secure; HttpOnly
x-content-type-options:  nosniff
x-debug-info:            eyJyZXRyaWVzIjowfQ==
x-frame-options:         SAMEORIGIN
x-platform-server:       i-087261e92df9f2aed
x-request-id:            hr74djxfx23r74mjhlizn6df
x-xss-protection:        1; mode=block
accept-ranges:           bytes
age:                     0
accept-ranges:           bytes
age:                     0
accept-ranges:           bytes
age:                     0
accept-ranges:           bytes
date:                    Sun, 28 Jul 2019 10:26:43 GMT
age:                     0
set-cookie:              _pxhd=518bb81822ce9d7eb4bcf241dbe060d923c5abf748f6f70a5fedc5ad70900f52:315a59c0-b122-11e9-b5de-590cfe45bdb4; Expires=Fri, 01 Jan 2021 00:00:00 GMT; path=/;
x-served-by:             cache-lhr6326-LHR, cache-akl1426-AKL
x-cache:                 MISS, MISS
x-cache-hits:            0, 0
vary:                    Accept-Encoding
content-length:          79
qwerty32123 commented 5 years ago

dunno why sends headers in very very strange way, for each "," in headers dict it sends in new value insttead of in the same, and my requests gets rejected cause of these, if sends all in same field like normal requests would be much better

VeNoMouS commented 5 years ago

@GnubiBORED not to disrespect this project, but i've moved onto httpx it seems more active and it aligns with requests model which is what i like and prefer, while alpha, i think it'll go places imho

qwerty32123 commented 5 years ago

@VeNoMouS i can speak with you somewhere? telegram? discord?

btw does httpx has support for proxy ? i cannot find it in their docs

VeNoMouS commented 5 years ago

@GnubiBORED it does not as of yet, but their dev's are responding to tickets, my discord is VeNoMouSNZ#5979

zb14 commented 5 years ago

I had the same problem. `

user-agent: | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML user-agent: | like Gecko) Chrome/76.0.3809.100 Safari/537.36 accept-encoding: | gzip accept-encoding: | deflate accept-encoding: | br

`

qwerty32123 commented 5 years ago

I had the same problem. `

user-agent: | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML user-agent: | like Gecko) Chrome/76.0.3809.100 Safari/537.36 accept-encoding: | gzip accept-encoding: | deflate accept-encoding: | br

`

move to httpx but it dosnt have proxy support

zb14 commented 5 years ago

I had the same problem. user-agent: | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML user-agent: | like Gecko) Chrome/76.0.3809.100 Safari/537.36 accept-encoding: | gzip accept-encoding: | deflate accept-encoding: | br

move to httpx but it dosnt have proxy support

hyper version 0.7.0 is ok, but it also doesn't have proxy support

sethmlarson commented 5 years ago

Just a BTW since this thread mentions proxy support for HTTPX a lot: We're working on proxy support currently. :) Would love to have some users to test it once PRs are made, can I cc everyone in this thread when that happens?

VeNoMouS commented 5 years ago

@sethmlarson whats a rough eta of like proxies and ssl context's being implemented into the current release? we all have tickets... but we dont know the time schedule?

I've been waiting 3 weeks for ssl contexts for example.

sethmlarson commented 5 years ago

Yeah the timing is unfortunate because the last two weeks I've changed job, moved to a new apartment, and traveled out of town. Not a lot of time for open source!

@tomchristie has also been on vacation for two weeks so larger features don't have reviewers and we haven't had a release recently. I've done a good amount of work on HTTP tunnel proxies. I can get the SSLContext change in quick if that is an issue for you.

Just keep in mind that the more contributors we have the faster features can be implemented. :)

qwerty32123 commented 5 years ago

Just a BTW since this thread mentions proxy support for HTTPX a lot: We're working on proxy support currently. :) Would love to have some users to test it once PRs are made, can I cc everyone in this thread when that happens?

when it happens you can for sure send me or send me a message in telegram if you want(@gnubie) im more active there anyway, i would like to test i have a big project that needs http/2 so yes :)

sethmlarson commented 5 years ago

BTW the verify=SSLContext() interface for SSLConfig is functional in master.

qwerty32123 commented 5 years ago

BTW the verify=SSLContext() interface for SSLConfig is functional in master.

In what way you do it? Im currently managing tls and ssl like these with requests, but these will not work in httpx i think


from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager
import ssl

class MyAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       ssl_version=ssl.PROTOCOL_TLSv1_2)`

then i just mount these into request session

session = requests.session()

session.mount('https://', MyAdapter())

sethmlarson commented 5 years ago

Actually in master TLSv1.2 and TLSv1.3 are already the only available protocols. :)

But let's say you only want ECDHE+AESGCM for ciphers you can do:

import httpx, ssl

ctx = ssl.SSLContext()
ctx.set_ciphers("ECDHE+AESGCM")
client = httpx.Client(verify=ctx)

...
qwerty32123 commented 5 years ago

i just need 1.2 so its fine. thanks for fast reply :)

sethmlarson commented 4 years ago

Just an FYI to everyone in this thread, HTTPX now ships with HTTP proxy support. :)

qwerty32123 commented 4 years ago

Just an FYI to everyone in this thread, HTTPX now ships with HTTP proxy support. :)

Its already testeable? or will come up in some days

sethmlarson commented 4 years ago

HTTPX 0.7.3 supports the proxies parameter and HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY environment variables. It should be testable so any defects you find should be reported as bugs. :)

See documentation for more info: https://encode.io/httpx/advanced

txtsd commented 4 years ago

This is happening at this line https://github.com/python-hyper/hyper/blob/18b629b8487169870235fe13387e2ae03c178b9f/hyper/common/headers.py#L252