Separating Requests from Responses

[anecdata]

I have yet to get my head fully around the Requests library, so this is a bit of a shot in the dark but I wonder if there is a way to more cleanly separate the Request from the Response, so that:

1. a Request could be issued
2. optionally, at a time of the user code's choosing, it could non-destructively peek ahead to see if there is any returned data yet on the socket or just do some other quick processing before proceeding to the Response
3. then at its leisure, it could read the status code and reason, response headers, and response content

This could potentially significantly reduce the blocking duration of a Request. Ideally, there is no change to existing user code, which typically issues the Request and immediately reads the Response. For those who want to take advantage of the separation, there could be some check for data between Request and Response, or at least an opportunity to use the wait time for some other operation.

One of the issues I see in this current library is that retries in Request require some reading of the Response. But I don't see that CPython does automatic retries (or reads ahead into the Response as part of the Request) - there's a lower-level class that allows a retries parameter which defaults to zero: class requests.adapters.HTTPAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False. The explanatory text for those retries is:

max_retries – The maximum number of retries each connection should attempt. Note, this applies only to failed DNS lookups, socket connections and connection timeouts, never to requests where data has made it to the server. By default, Requests does not retry failed connections. If you need granular control over the conditions under which we retry a request, import urllib3’s Retry class and pass that instead.

This comment seems to imply a clean separation of Request and Response:

a Response object is generated once Requests gets a response back from the server

This all may become moot if we get some kind of pre-emptive async.

[anecdata]

Bumping this for visibility in light of pending asyncio feature. My understanding is that asyncio is cooperative rather than pre-emptive, so Requests would need to be restructured to take advantage of it.

[tannewt]

I don't think we'd change requests. Instead, we'd provide a second async library, just like aiohttp in CPython. https://www.twilio.com/blog/asynchronous-http-requests-in-python-with-aiohttp

[anecdata]

Closing in favor of a separate future async-friendly library.