Support for iterating over very large responses

[GoogleCodeExporter]

In httplib, one can read the response body iteratively so he needs only a
fixed amount of memory for very large responses. For example, in an WSGI
app one can write:

    ...
    size = 8192
    with closing(conn):
        res = conn.getresponse()
        content = iter(lambda: res.read(size), '')
        start_response('200 OK', [])
        return content

It would be nice to have such a possibility in httplib2.

Original issue reported on code.google.com by anrien...@gmail.com on 8 May 2008 at 2:04

• Merged into: #5

[GoogleCodeExporter]

Oops, I've just realized that in this example I have to have the connection 
opened
until all the response data is read. Anyway, `with closing(conn)` isn't 
important in
this context.

Original comment by anrien...@gmail.com on 8 May 2008 at 2:09

[GoogleCodeExporter]

This is harder than it sounds because that body might need to be unzipped or
decompressed in chunks, and also interacts with caching, not that these are
insurmountable obstacles. Will keep this open as an enhancement request.

Original comment by joe.gregorio@gmail.com on 7 Sep 2008 at 10:42

• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Is there any patch available on this issue?

Original comment by santoz.2...@gmail.com on 29 Sep 2009 at 6:40

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

HTTP documents can be arbitrarily huge, often much bigger than the memory 
available
(think downloading a video on a memory restricted device). Often we want to 
download
stuff directly to disk (and not create an object in memory).

Original comment by salimfadhley@gmail.com on 1 Dec 2009 at 12:20

[GoogleCodeExporter]

Original comment by joe.gregorio@gmail.com on 26 Dec 2009 at 5:09

• Changed state: Duplicate