[uvloop] aiohttp client and proxy - TimeoutError

[estin]

When fetch url with proxy and uvloop like this:

import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

import aiohttp

async def fetch(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            return await response.text()

async def main_with_proxy(loop):
    conn = aiohttp.ProxyConnector(proxy="http://127.0.0.1:1091")
    async with aiohttp.ClientSession(connector=conn, loop=loop) as session:
        html = await fetch(session, 'http://python.org')
        print(html)

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main_with_proxy(loop))

Result:

Traceback (most recent call last):
  File "uvloop-test.py", line 26, in <module>
    loop.run_until_complete(main_with_proxy(loop))
  File "uvloop/loop.pyx", line 1133, in uvloop.loop.Loop.run_until_complete (uvloop/loop.c:19911)
  File "uvloop/future.pyx", line 123, in uvloop.loop.BaseFuture.result (uvloop/loop.c:93421)
  File "uvloop/future.pyx", line 78, in uvloop.loop.BaseFuture._result_impl (uvloop/loop.c:92960)
  File "uvloop/task.pyx", line 128, in uvloop.loop.BaseTask._fast_step (uvloop/loop.c:98739)
  File "uvloop-test.py", line 15, in main_with_proxy
    html = await fetch(session, 'http://python.org')
  File "uvloop-test.py", line 10, in fetch
    return await response.text()
  File "/root/lib/python3.5/site-packages/aiohttp-0.22.5-py3.5-linux-x86_64.egg/aiohttp/helpers.py", line 564, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

But on same proxy with curl:

$ curl -v --proxy 127.0.0.1:1091 --url http://python.org/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 1091 (#0)
> GET http://python.org/ HTTP/1.1
> User-Agent: curl/7.38.0
> Host: python.org
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 301 Moved Permanently
< DeleGate-Ver: 9.9.13 (delay=0)
< Location: https://python.org/
< Content-length: 0
< Via: 1.1 - (DeleGate/9.9.13)
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
< 
* Closing connection 0

Environment

Python 3.5.1
aiohttp 0.22.5
uvloop 0.5.1

Where

• aiohttp with uvloop, but without proxy works
• aiohttp with proxy, but wihout uvloop works

[asvetlov]

I really have no idea what's going wrong. aiohttp proxy has no magic, if it works with standard asyncio loop -- it should work with uvloop as well.

[asvetlov]

@1st1 Maybe as uvloop author you might add a comment

[estin]

With allow_redirects=False

import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

import aiohttp

async def fetch(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url, allow_redirects=False) as response:
            print('Response:', response)
            return await response.text()

async def main_with_proxy(loop):
    conn = aiohttp.ProxyConnector(proxy="http://127.0.0.1:1091", loop=loop)
    async with aiohttp.ClientSession(connector=conn, loop=loop) as session:
        body = await fetch(session, 'http://python.org')
        print('Body len:', len(body))

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main_with_proxy(loop))

Result - OK

Response: <ClientResponse(http://python.org/) [301 Moved Permanently]>
<CIMultiDictProxy('DELEGATE-VER': '9.9.13 (delay=0)', 'LOCATION': 'https://python.org/', 'CONTENT-LENGTH': '0', 'VIA': '1.1 - (DeleGate/9.9.13)', 'CONNECTION': 'close')>

Body len: 0

I think something wrong on ssl context.

[asvetlov]

Hmm. @estin are you able to fetch https://python.org/ without proxy but using uvloop?

[estin]

Hmm. @estin are you able to fetch https://python.org/ without proxy but using uvloop?

Yes. Results below

Fetch https://python.org without proxy but using uvloop

----------
RESULT:  OK
allow_redirects: True
url: https://python.org
with_proxy: False

----------
RESULT:  OK
allow_redirects: False
url: https://python.org
with_proxy: False

Do not forget https://python.org/ redirects to https://www.python.org/

$ curl -D - --proxy 127.0.0.1:1090 --url https://python.org/
HTTP/1.1 200 Connection established.
Proxy-Connection: close
Proxy-Agent: DeleGate/9.9.13

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Thu, 11 Aug 2016 09:18:15 GMT
Content-Type: text/html
Content-Length: 178
Location: https://www.python.org/
Strict-Transport-Security: max-age=315360000; preload

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

Test case

import os
import asyncio

if int(os.environ.get("USE_UVLOOP", 0)):
    print("with uvloop")
    import uvloop
    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

import aiohttp

async def fetch(session, allow_redirects, url):
    with aiohttp.Timeout(10):
        async with session.get(url, allow_redirects=allow_redirects) as response:  # noqa
            return await response.text()

async def main_with_proxy(loop, with_proxy, allow_redirects, url):
    conn = None
    if with_proxy:
        conn = aiohttp.ProxyConnector(proxy="http://127.0.0.1:1090", loop=loop)
    async with aiohttp.ClientSession(connector=conn, loop=loop) as session:
        body = await fetch(session, allow_redirects, url)

async def test(loop):

    TO_TEST = (
        # https
        {
            'with_proxy': True,
            'allow_redirects': True,
            'url': 'https://python.org',
        },
        {
            'with_proxy': True,
            'allow_redirects': False,
            'url': 'https://python.org',
        },
        {
            'with_proxy': False,
            'allow_redirects': True,
            'url': 'https://python.org',
        },
        {
            'with_proxy': False,
            'allow_redirects': False,
            'url': 'https://python.org',
        },

        # http -> https
        {
            'with_proxy': True,
            'allow_redirects': True,
            'url': 'http://python.org',
        },
        {
            'with_proxy': True,
            'allow_redirects': False,
            'url': 'http://python.org',
        },
        {
            'with_proxy': False,
            'allow_redirects': True,
            'url': 'http://python.org',
        },
        {
            'with_proxy': False,
            'allow_redirects': False,
            'url': 'http://python.org',
        },
    )

    for params in TO_TEST:
        try:
            await main_with_proxy(loop, **params)
            params.update({
                'result': 'OK',
            })
        except Exception as e:
            params.update({
                'result': type(e).__name__,
            })

    for test in TO_TEST:
        print()
        print('-' * 10)
        print('RESULT: ', test.pop('result'))
        for k, v in test.items():
            print('{}: {}'.format(k, v))

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(test(loop)) 

Without uvloop - OK

$ USE_UVLOOP=0 python uvloop-test.py 

----------
RESULT:  OK
allow_redirects: True
url: https://python.org
with_proxy: True

----------
RESULT:  OK
allow_redirects: False
url: https://python.org
with_proxy: True

----------
RESULT:  OK
allow_redirects: True
url: https://python.org
with_proxy: False

----------
RESULT:  OK
allow_redirects: False
url: https://python.org
with_proxy: False

----------
RESULT:  OK
allow_redirects: True
url: http://python.org
with_proxy: True

----------
RESULT:  OK
allow_redirects: False
url: http://python.org
with_proxy: True

----------
RESULT:  OK
allow_redirects: True
url: http://python.org
with_proxy: False

----------
RESULT:  OK
allow_redirects: False
url: http://python.org
with_proxy: False

With uvloop

$ USE_UVLOOP=1 python uvloop-test.py
with uvloop

----------
RESULT:  TimeoutError
allow_redirects: True
url: https://python.org
with_proxy: True

----------
RESULT:  TimeoutError
allow_redirects: False
url: https://python.org
with_proxy: True

----------
RESULT:  OK
allow_redirects: True
url: https://python.org
with_proxy: False

----------
RESULT:  OK
allow_redirects: False
url: https://python.org
with_proxy: False

----------
RESULT:  TimeoutError
allow_redirects: True
url: http://python.org
with_proxy: True

----------
RESULT:  OK
allow_redirects: False
url: http://python.org
with_proxy: True

----------
RESULT:  OK
allow_redirects: True
url: http://python.org
with_proxy: False

----------
RESULT:  OK
allow_redirects: False
url: http://python.org
with_proxy: False

[asvetlov]

Thanks you for comprehensive test case!

[1st1]

Hi, thanks for reporting this. What kind of proxy do I need to use to reproduce this?

[1st1]

I assume the used proxy is DeleGate v9.9.13. I can't reproduce TimeoutError on either MacOS or Linux.

[1st1]

What version of uvloop are you using? What platform are you running the script on? What version of Python? Of aiohttp? etc.

[estin]

Hi, thanks for reporting this. What kind of proxy do I need to use to reproduce this?

Any http proxy.

I have same behavior on

• squide
• delegate

Delegate http://www.delegate.org/delegate/

$ wget http://delegate.hpcc.jp/anonftp/DeleGate/bin/linux/latest/linux2.6-dg9_9_13.tar.gz
$ tar xzf linux2.6-dg9_9_13.tar.gz
$ ./dg9_9_13/DGROOT/bin/dg9_9_13 -P127.0.0.1:1090 SERVER=http ADMIN=fake@fake.com

and then run test case from https://github.com/KeepSafe/aiohttp/issues/1065#issuecomment-239111719

What version of uvloop are you using? What platform are you running the script on? What version of Python? Of aiohttp? etc.

Inside docker container from https://hub.docker.com/_/python/

root@b408a24eb441:~# uname -a
Linux b408a24eb441 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64 GNU/Linux

root@b408a24eb441:~# python -V
Python 3.5.1

root@b408a24eb441:~# pip freeze | egrep '(aiohttp|uvloop)'
aiohttp==0.22.5
uvloop==0.5.1

root@b408a24eb441:~# autoconf -V
autoconf (GNU Autoconf) 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+/Autoconf: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>, <http://gnu.org/licenses/exceptions.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by David J. MacKenzie and Akim Demaille.

root@b408a24eb441:~# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10) 

[estin]

100% case to reproduce TimeoutError for me

Run docker container

docker run --rm -it python:latest bash

Inside docker container

wget http://delegate.hpcc.jp/anonftp/DeleGate/bin/linux/latest/linux2.6-dg9_9_13.tar.gz
tar xzf linux2.6-dg9_9_13.tar.gz
./dg9_9_13/DGROOT/bin/dg9_9_13 -P127.0.0.1:1090 SERVER=http ADMIN=fake@fake.com
pip install aiohttp uvloop
wget https://gist.githubusercontent.com/estin/a582d83bae105016094b5a76516663a9/raw/63113e6c3added58fd809e47f03f6a38646c5a4d/uvloop-test.py
USE_UVLOOP=1 python uvloop-test.py 

Output

with uvloop

----------
RESULT:  TimeoutError
url: https://python.org
with_proxy: True
allow_redirects: True

----------
RESULT:  TimeoutError
url: https://python.org
with_proxy: True
allow_redirects: False

----------
RESULT:  OK
url: https://python.org
with_proxy: False
allow_redirects: True

----------
RESULT:  OK
url: https://python.org
with_proxy: False
allow_redirects: False

----------
RESULT:  TimeoutError
url: http://python.org
with_proxy: True
allow_redirects: True

----------
RESULT:  OK
url: http://python.org
with_proxy: True
allow_redirects: False

----------
RESULT:  OK
url: http://python.org
with_proxy: False
allow_redirects: True

----------
RESULT:  OK
url: http://python.org
with_proxy: False
allow_redirects: False

Host machine environment

$ uname -a
Linux localhost 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64 GNU/Linux

$ docker --version
Docker version 1.11.2, build b9f10c9

[1st1]

This is now fixed in uvloop v0.5.2. Thanks a bunch for reporting this issue.

[estin]

Thanks a lot!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

If you feel like there's important points made in this discussion, please include those exceprts into that new issue.