search

encode / httpx

A next generation HTTP client for Python. 🦋
https://www.python-httpx.org/
BSD 3-Clause "New" or "Revised" License
13.03k stars 829 forks source link

httpx.ReadError is raised with httpx async api when requesting certain insecure ssl website #1896

Closed tomchristie closed 2 years ago

tomchristie commented 2 years ago

I can reproduce discussion #1842, in a very simple example:

async def main():
    async with httpx.AsyncClient() as client:
        r = await client.get('https://www.comdirect.de/')
        print(r)
        r = await client.get('https://www.comdirect.de/')
        print(r)

if __name__ == '__main__':
    asyncio.run(main())

Which raises an httpx.ReadError()

full traceback ```python Traceback (most recent call last): File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/anyio/streams/tls.py", line 102, in _call_sslobject_method result = func(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 718, in read v = self._sslobj.read(len) ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2555) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_backends/anyio.py", line 60, in read return await self.stream.receive(n) File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/anyio/streams/tls.py", line 152, in receive data = await self._call_sslobject_method(self._ssl_object.read, max_bytes) File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/anyio/streams/tls.py", line 121, in _call_sslobject_method raise BrokenResourceError from exc anyio.BrokenResourceError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/tomchristie/GitHub/encode/httpx/httpx/_transports/default.py", line 62, in map_httpcore_exceptions yield File "/Users/tomchristie/GitHub/encode/httpx/httpx/_transports/default.py", line 201, in __aiter__ async for part in self._httpcore_stream: File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 57, in __aiter__ async for chunk in self.stream: File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_bytestreams.py", line 91, in __aiter__ async for chunk in self._aiterator: File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 208, in _receive_response_data event = await self._receive_event(timeout) File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 225, in _receive_event data = await self.socket.read(self.READ_NUM_BYTES, timeout) File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_backends/anyio.py", line 65, in read raise ReadError from exc httpcore.ReadError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "./example.py", line 46, in asyncio.run(main()) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete return future.result() File "./example.py", line 40, in main r = await client.get('https://www.comdirect.de/') File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1712, in get timeout=timeout, File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1484, in request request, auth=auth, follow_redirects=follow_redirects File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1585, in send raise exc File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1579, in send await response.aread() File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1662, in aread self._content = b"".join([part async for part in self.aiter_bytes()]) File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1662, in self._content = b"".join([part async for part in self.aiter_bytes()]) File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1678, in aiter_bytes async for raw_bytes in self.aiter_raw(): File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1732, in aiter_raw async for raw_stream_bytes in self.stream: File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 147, in __aiter__ async for chunk in self._stream: File "/Users/tomchristie/GitHub/encode/httpx/httpx/_transports/default.py", line 202, in __aiter__ yield part File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__ self.gen.throw(type, value, traceback) File "/Users/tomchristie/GitHub/encode/httpx/httpx/_transports/default.py", line 79, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ReadError ```

And which is different behaviour from the sync equivalent.

I haven't managed to dig into why the SSL behaviour is different in the two cases here, but it's clearly something worth looking at in more detail.

tomchristie commented 2 years ago

Right, I've tracked this one down.

See also https://github.com/encode/httpcore/issues/396 which is the same thing again.

I'l open an issue against anyio to deal with this case.

tomchristie commented 2 years ago

Updating to the latest version of anyio should now resolve this. 👍

Thanks for the report. 😊