Open njsmith opened 5 years ago
Had more discussions about this, here's the results of that discussion:
Response.as_file()
method which exposes a file-like object that acts exactly as a usual open()
file is desirable for csv readers, data-frames, etcIOBase
but doesn't exactly behave like a file. Let's not make that mistake again.as_file()
interface shouldn't close once all data is read just like a real file but should close the contained Response object after all data is read. This prevents Responses from being left un-closed but also gives an interface that places nice with interfaces expecting files.Also discussed the usage of .text()/.data()/.json() as methods that load all data onto the Response.
There were questions about whether you should be able to call .text()
and then maybe .text()
again.
If we are one-shot only then you'd only be able to call a single "body" function on a response. However @pquentin brought up that this would make things annoying for debugging as you'd have to reissue requests to get another body.
Another thought was that if a response body is completely loaded into memory we could hold onto that body internally on the request to allow calling .text(), .json(), and .data() again if needed.
In urllib3,
HTTPResponse
inherits fromio.IOBase
and implements the standard Python file interface. This is an interesting issue that discusses a bug in their implementation, and also why people value this feature: https://github.com/urllib3/urllib3/issues/1305We need to decide what interface our version of
HTTPResponse
should expose. Constraints:IOBase
, becauseIOBase
is a sync interface, and for async we probably want to expose something compatible with trio'sReceiveStream
APIIOBase
, so people can keeping feeding it into thecsv
module etc., like they're doing in the issue I linked aboveasync
markersYou'll notice that these constraints are contradictory :-). Especially since the
ReceiveStream
andIOBase
interfaces actually conflict with each other: they both support iteration, but forReceiveStream
it gives arbitrary chunks of bytes, while forIOBase
it gives lines.Some options:
HTTPResponse
implement theIOBase
API in sync mode, and theReceiveStream
API in async mode.IOBase
in sync mode, and an async version ofIOBase
in async mode, and also provide anas_stream()
methodReceiveStream
in async mode, a sync-ified version ofReceiveStream
in sync mode, and also provide anas_file()
method.as_file
method could either be not defined at all, or it could still exist and return an async version of the file interfaceas_stream
andas_file
methods (and then there's the same question about which methods are available in which modes)