daijro / hrequests

🚀 Web scraping for humans
https://daijro.gitbook.io/hrequests/
MIT License
654 stars 39 forks source link

Can we use auth proxies? #28

Closed cosmicsplendor closed 6 months ago

cosmicsplendor commented 10 months ago

How could we accomplish something like this using hrequests?

import requests

proxies = {
   'http': 'http://proxy.example.com:8080',
   'https': 'http://proxy.example.com:8081',
}

response = requests.get('http://httpbin.org/ip', proxies=proxies auth=('USERNAME', 'PASSWORD'))
cosmicsplendor commented 9 months ago

This library supports normal http proxies (example: http://proxy.example.com:8080).

But I couldn't get it to work with proxies that require username password authentication (example: http://username:password@proxy.example.com:8080).

I'd appreciate if you could me know whether and how this could be achieved.

Cosmicoppai commented 9 months ago

@cosmicsplendor hey, are able to find any workaround with this lib?

ilyazub commented 9 months ago

Related to https://github.com/bogdanfinn/tls-client/issues/66.

cosmicsplendor commented 9 months ago

@cosmicsplendor hey, are able to find any workaround with this lib?

I haven't been able to get this to work yet. Will update here if I find a workaround.

cosmicsplendor commented 9 months ago

this works, but you have to rely on requests library to create session:

import requests import hrequests

proxies = { 'http': 'http://USERNAME:PASSWORD@proxy.example.com', 'https': 'http://USERNAME:PASSWORD@proxy.example.com' }

session = requests.Session() session.proxies.update(proxies)

response = hrequests.get('http://httpbin.org/ip', session=session) print(response.text)

ilyazub commented 9 months ago

@cosmicsplendor What hrequests version do you use? It worked for me with v0.7.1.

>>> import hrequests
>>> response = hrequests.get('http://httpbin.org/ip', debug=True, verify=False, proxies=proxies)
raw request bytes sent over wire: 344 (0 kb)
headers on request:
map[Accept:[*/*] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US;q=0.5,en;q=0.3] Connection:[keep-alive] Header-Order::[] Host:[httpbin.org] Referer:[https://google.com] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.6167.0 Safari/537.36]]
cookies on request:
[]
headers on response:
map[Access-Control-Allow-Credentials:[true] Access-Control-Allow-Origin:[*] Connection:[keep-alive] Content-Length:[31] Content-Type:[application/json] Date:[Wed, 10 Jan 2024 10:41:27 GMT] Server:[gunicorn/19.9.0]]
cookies on response:
[]
requested http://httpbin.org/ip : status 200
response body payload: {
  "origin": "...omitted..."
}

raw response bytes received over wire: 260 (0 kb)
>>> hrequests.__version__
'0.7.1'
cosmicsplendor commented 9 months ago

Mine is v0.7.1 too. I tried hard, but could not make it work. Normal proxies work fine, but when I try authentication, it always fails.

HOSEENJ commented 6 months ago

to use proxy in sessions you can do like this: session = hrequests.Session() session.proxy = 'http://username:password@host:port'