search

aio-libs / aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
https://docs.aiohttp.org
Other
14.95k stars 1.99k forks source link

Unable to use HTTP proxies #4719

Closed Th0rgal closed 2 weeks ago

Th0rgal commented 4 years ago

🐞 Describe the bug Hi, I hope I'm not bothering you with a bug that is actually a problem of use on my part but it's been several days that I've been trying by all means to use proxies with AioHTTP but I can't manage to do so. My goal is to make a small client for my app and it must absolutely have an option to be anonymous (through a proxy) because it is very privacy oriented. For my tests I'm just trying to get the contents of the page of one of my other projects (it's called hackedserver, but don't worry, it's a teaser name and it doesn't have much to do with security). When I'm not using a proxy it works great, no worries. But when I use one, I make a big error message.

💡 To Reproduce You can just try my demo code:

import aiohttp
import asyncio

async def test():
    async with aiohttp.ClientSession() as session:
        async with session.request('GET', "https://hackedserver.info/", proxy = "http://80.187.140.26:8080") as response:
                print("resp:" + str(response))

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(test())
    loop.run_forever()

I am using Python 3.8 with AIoHTTP 3.6.2 (last version if I'm not wrong)

💡 Expected behavior Here is what I get without a proxy:

resp:<ClientResponse(https://hackedserver.info/) [200 OK]>
<CIMultiDictProxy('Server': 'nginx', 'Date': 'Sun, 26 Apr 2020 20:30:20 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Last-Modified': 'Thu, 05 Mar 2020 19:00:23 GMT', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'ETag': 'W/"5e614c47-3967"', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Content-Encoding': 'gzip')>

📋 Logs/tracebacks

Traceback (most recent call last):
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\proactor_events.py", line 395, in _loop_writing
    self._write_fut = self._loop._proactor.send(self._sock, data)
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 525, in send
    self._register_with_iocp(conn)
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 714, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] Paramètre incorrect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".\test.py", line 11, in <module>
    loop.run_until_complete(test())
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File ".\test.py", line 6, in test
    async with session.request('GET', "https://hackedserver.info/", proxy = "http://80.187.140.26:8080") as response:
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\client.py", line 480, in _request
    conn = await self._connector.connect(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 855, in _create_connection
    _, proto = await self._create_proxy_connection(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 1093, in _create_proxy_connection
    transport, proto = await self._wrap_create_connection(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host hackedserver.info:443 ssl:default [Paramètre incorrect]

It says that a parameter is incorrect (I translate because it is written in french). I don't understand what parameter could be incorrect.

📋 Your version of the Python

Python 3.8.2

📋 Your version of the aiohttp/yarl/multidict distributions

python -m pip show aiohttp 3.6.2
multidict 4.7.5
yarl 1.4.2

📋 Additional context I was using aiohttp (client) on windows 10 (64 bits). I can also try on nixos 20.03 if needed (on my laptop).

webknjaz commented 4 years ago

Please check: 1) if you run aiohttp with Cython extensions or without them (pure-python) 2) if running through 64-bit Python helps (your log suggests that you use a 32-bit version) 3) if using Linux help (with or without C-extensions) 4) if your proxy supports connections to HTTPS resources and whether it works with some other tool like curl or your browsers

Th0rgal commented 4 years ago

1) I'm running a newly installed python with default settings; I believe this is pure python 2) I reinstalled it in 64 bits, same issue:

Traceback (most recent call last):
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 395, in _loop_writing
    self._write_fut = self._loop._proactor.send(self._sock, data)
  File "C:\Program Files\Python38\lib\asyncio\windows_events.py", line 525, in send
    self._register_with_iocp(conn)
  File "C:\Program Files\Python38\lib\asyncio\windows_events.py", line 714, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] Paramètre incorrect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".\test.py", line 11, in <module>
    loop.run_until_complete(test())
  File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File ".\test.py", line 6, in test
    async with session.request('GET', "https://hackedserver.info/", proxy = "http://80.187.140.26:8080") as response:
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\client.py", line 480, in _request
    conn = await self._connector.connect(
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 855, in _create_connection
    _, proto = await self._create_proxy_connection(
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 1093, in _create_proxy_connection
    transport, proto = await self._wrap_create_connection(
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host hackedserver.info:443 ssl:default [Paramètre incorrect]
  1. Linux help: I don't think so. How to check?
  2. I'll try with a chrome extension.

EDIT: So I tried with foxy proxy and this proxy works well with hackedserver.info (and this is with HTTPS). I can't use google though (privacy error).

webknjaz commented 4 years ago

(1) & (3): you need to pay attention to your pip install to find out what you install.

Th0rgal commented 4 years ago

(1) & (3): you need to pay attention to your pip install to find out what you install.

C:\Windows\system32>pip3 list
Package           Version
----------------- ---------
aiohttp           3.6.2
altgraph          0.17
appdirs           1.4.3
astroid           2.3.3
async-timeout     3.0.1
attrs             19.3.0
black             19.10b0
chardet           3.0.4
click             7.1.1
colorama          0.4.3
entrypoints       0.3
flake8            3.7.9
future            0.18.2
idna              2.9
isort             4.3.21
lazy-object-proxy 1.4.3
mccabe            0.6.1
multidict         4.7.5
pathspec          0.8.0
pefile            2019.4.18
pip               20.0.2
pycodestyle       2.5.0
pyflakes          2.1.1
PyInstaller       3.6
pylint            2.4.4
pywin32-ctypes    0.2.0
regex             2020.4.4
setuptools        41.2.0
six               1.14.0
toml              0.10.0
typed-ast         1.4.1
wrapt             1.11.2
yarl              1.4.2

Is that what you need?

webknjaz commented 4 years ago

No, it's only visible in the log of the pip install command

Th0rgal commented 4 years ago

No, it's only visible in the log of the pip install command

I'm probably dumb, but when I type pip install I get this:

C:\Windows\system32>pip3 install
ERROR: You must give at least one requirement to install (see "pip help install")

sorry

webknjaz commented 4 years ago

@Th0rgal I mean pip install aiohttp, of course :)

Th0rgal commented 4 years ago

@Th0rgal I mean pip install aiohttp, of course :)

Oh okay, I'm sorry. Here is the result:

C:\Windows\system32>pip3 install aiohttp             
Collecting aiohttp        
Using cached aiohttp-3.6.2-py3-none-any.whl (441 kB)
Requirement already satisfied: yarl<2.0,>=1.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (1.4.2)
Requirement already satisfied: attrs>=17.3.0 in c:\program files\python38\lib\site-packages (from aiohttp) (19.3.0)
Requirement already satisfied: multidict<5.0,>=4.5 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (4.7.5)
Requirement already satisfied: async-timeout<4.0,>=3.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (3.0.1) 
Requirement already satisfied: chardet<4.0,>=2.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (3.0.4)  
Requirement already satisfied: idna>=2.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from yarl<2.0,>=1.0->aiohttp) (2.9)   
Installing collected packages: aiohttp                                      
Successfully installed aiohttp-3.6.2
webknjaz commented 4 years ago

Using cached aiohttp-3.6.2-py3-none-any.whl (441 kB)

So it uses a pure-python wheel, no C-extensions. Which is sort of expected (https://pypi.org/project/aiohttp/3.6.2/#files doesn't have wheels for Python 3.8). You could force it to build from source tho.

Try out Linux, it'll probably pick up a manylinux wheel unless you use Python 3.8 there too.

Th0rgal commented 4 years ago

Alright. So this is working on my linux environment:

✘ thomas@horus  ~  python ./test.py
resp:<ClientResponse(https://hackedserver.info/) [200 OK]>
<CIMultiDictProxy('Server': 'nginx', 'Date': 'Mon, 27 Apr 2020 13:08:43 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Last-Modified': 'Thu, 05 Mar 2020 19:00:23 GMT', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Etag': 'W/"5e614c47-3967"', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Content-Encoding': 'gzip')>

Linux: Nixos 20.03 (kernel 5.5.17) Python 3.7.6 aiohttp 3.6.2 multidict 4.7.4 yarl 1.4.2

webknjaz commented 4 years ago

@Th0rgal py3.7 probably picked up the version with C-extensions so the logic there is probably different. Also, this may be some limitation of the asyncio loop on Windows

webknjaz commented 4 years ago

This may be a bug in CPython: https://superuser.com/a/1474491/182499

Th0rgal commented 4 years ago

Do you think I could get it to work on windows using Python 3.7.6? Should I specify something to use C-extensions?

webknjaz commented 4 years ago

Should I specify something to use C-extensions?

You can force pip use sdist but then you'd need the build toolchain on your machine.

Do you think I could get it to work on windows using Python 3.7.6?

Maybe, I'm not sure. It looks like a Proactor bug: https://github.com/tiangolo/fastapi/issues/759.

Th0rgal commented 4 years ago

Alright so I installed Python 3.7.6 on windows and this is now working fine. Since this is not really a aiohttp bug, do you want me to close the issue? Thanks for your great help!

webknjaz commented 4 years ago

Let's keep it for now. I'm still not sure if we can have some workaround for this in aiohttp.

WH-2099 commented 4 years ago

I got the same problem, too. I think it's just like the issue #2245. And the solution mentioned could help.

Changed in version 3.8: On Windows, ProactorEventLoop is now the default event loop. https://docs.python.org/3/library/asyncio-platforms.html

ProactorEventLoop is default in Python 3.8 on Windows which breaks proxy support for https urls.

Solution:

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
dustiwe commented 3 years ago

I got the same problem, too. I think it's just like the issue #2245. And the solution mentioned could help.

Changed in version 3.8: On Windows, ProactorEventLoop is now the default event loop. https://docs.python.org/3/library/asyncio-platforms.html

ProactorEventLoop is default in Python 3.8 on Windows which breaks proxy support for https urls. Solution:

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

THANK YOU !!!!!!!!!!!❤❤❤

isidentical commented 3 years ago

This seems like the issue on #4536, and I wonder whether this needs to be solved on the upstream or not?

munding commented 3 years ago

Why does this problem arise？what‘s ProactorEventLoop problem？

Dreamsorcerer commented 3 weeks ago

Is this still reproducible?

djhenya commented 2 weeks ago

Is this still reproducible?

Y

Dreamsorcerer commented 2 weeks ago

Is this still reproducible?

Y

Can someone provide me with an up-to-date reproducer? The original doesn't seem to connect to anything anymore.

djhenya commented 2 weeks ago

upd: Not reproducible with aiohttp==3.10.4