Is it possible to use an async dns resolver with requests-futures

[allan-simon]

Hello I'm evaluating requests-futures to be used inside a Tornado project, and I would like to know if the dns resolver used by requests futures is non-blocking (if this question is even relevant?)

[ross]

Provided the DNS lookup happens during the request call which I would assume to be the case. If so that would happen in the worker and thus not block. You could probably test pretty easily by setting your DNS resolvers to non-routable ip addresses and seeing if things block.

[guettli]

@allan-simon could you please look at this issue again? Did the reply of ross help you? What is your final result? Did you use requests-futures or a different solution ... I am curious since I have the same question. Please share your results. Thank you.

[allan-simon]

@guettli unfortunately we ended using directly the httpclient of tornado so we didn't push any further :(

[ross]

Got a chance to spend a couple minutes to test this out. As best as I can tell my initial guess was correct, that the dns resolution already happens in the background.

#!/usr/bin/env python

from requests_futures.sessions import FuturesSession
from time import time

from urllib3.util import connection

_orig_create_connection = connection.create_connection

def patched_create_connection(address, *args, **kwargs):
    """Wrap urllib3's create_connection to resolve the name elsewhere"""
    # resolve hostname to an ip address; use your own
    # resolver here, as otherwise the system resolver will be used.
    raise Exception('boom')

connection.create_connection = patched_create_connection

session = FuturesSession()
# first request is started in background
print('before', time())
future = session.get('http://httpbin.org/get')
print('future', time())
resp = future.result()
print('done', time())

(env)otter:tmp ross$ python dns.py
('before', 1518023857.001204)
('future', 1518023857.006673)
Traceback (most recent call last):
  File "dns.py", line 25, in <module>
    resp = future.result()
  File "/private/tmp/env/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462, in result
    return self.__get_result()
  File "/private/tmp/env/lib/python2.7/site-packages/concurrent/futures/thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/private/tmp/env/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/private/tmp/env/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/private/tmp/env/lib/python2.7/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/private/tmp/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/private/tmp/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 893, in _send_output
    self.send(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 855, in send
    self.connect()
  File "/private/tmp/env/lib/python2.7/site-packages/urllib3/connection.py", line 166, in connect
    conn = self._new_conn()
  File "/private/tmp/env/lib/python2.7/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "dns.py", line 16, in patched_create_connection
    raise Exception('boom')
Exception: boom

Note that the exception is thrown after control has returned to the main thread and printed out the future's time and that essentially no time has passed at that point.

Fwiw this was using the technique described in https://stackoverflow.com/questions/22609385/python-requests-library-define-specific-dns to do custom dns lookup. There's further info there about plugging custom stuff into requests, but that doesn't appear to be necessary here in terms of avoiding blocking.

[allan-simon]

thanks for the explanatory code :)