Closed didip closed 4 years ago
A bit of a 'Heisenbug', happens less than 10% of the time when requesting (many) JSON files for episode metadata from the BBC's radio schedules.
@lmmx - Could you provide enough information to reproduce this? I'd be happy to spend some time looking into it.
also answering 'yes' to any of these questions means that this is not a httpx problem:
BackgroundTask
s?BaseHTTPMiddleware
or middleware
decorator from Starlette?i have same problem with fastapi + uvicorn. can i use uwsgi replace uvicorn to fix it ?
I could reproduce with this snippet:
import asyncio
async def main():
async with AsyncClient() as session:
await asyncio.wait([
asyncio.create_task(session.get('http://localhost:80'))
for _ in range(1000)
])
asyncio.run(main())
the ConnectionClosed seems to be generated only in this place in h11:
def _extract_next_receive_event(self):
state = self.their_state
# We don't pause immediately when they enter DONE, because even in
# DONE state we can still process a ConnectionClosed() event. But
# if we have data in our buffer, then we definitely aren't getting
# a ConnectionClosed() immediately and we need to pause.
if state is DONE and self._receive_buffer:
return PAUSED
if state is MIGHT_SWITCH_PROTOCOL or state is SWITCHED_PROTOCOL:
return PAUSED
assert self._reader is not None
event = self._reader(self._receive_buffer)
if event is None:
if not self._receive_buffer and self._receive_buffer_closed:
# In some unusual cases (basically just HTTP/1.0 bodies), EOF
# triggers an actual protocol event; in that case, we want to
# return that event, and then the state will change and we'll
# get called again to generate the actual ConnectionClosed().
if hasattr(self._reader, "read_eof"):
event = self._reader.read_eof()
else:
event = ConnectionClosed()
if event is None:
event = NEED_DATA
return event
and _receive_buffer_closed seems to be closed only in
def receive_data(self, data):
if data:
if self._receive_buffer_closed:
raise RuntimeError("received close, then received more data?")
self._receive_buffer += data
else:
self._receive_buffer_closed = True
meaning it was called with an empty data
in turn, this comes from httpcore:
async def _receive_event(self, timeout: float = None) -> H11Event:
while True:
with map_exceptions({h11.RemoteProtocolError: RemoteProtocolError}):
event = self._h11_state.next_event()
if event is h11.NEED_DATA:
data = await self._network_stream.read(
self.READ_NUM_BYTES, timeout=timeout
)
self._h11_state.receive_data(data)
else:
return event
so _network_stream.read
returned empty data
as I'm using asyncio, it's an AsyncIOStream
with this code:
async def read(self, max_bytes: int, timeout: float = None) -> bytes:
exc_map = {
TimeoutError: ReadTimeout,
anyio.BrokenResourceError: ReadError,
}
with map_exceptions(exc_map):
with anyio.fail_after(timeout):
try:
return await self._stream.receive(max_bytes=max_bytes)
except anyio.EndOfStream: # pragma: nocover
return b""
so I guess we reached EndOfStream on the client side then emitted the transition (their_role=SERVER, state=SEND_RESPONSE, event_type=ConnectionClosed) which is not managed.
When programming on the server side, it means client reached ConnectionClosed while the server is the sending the response/client is receiving, I think the bug does not happen since the server would not be calling receive_data while sending the response.
On the client side, it means the server reached ConnectionClosed while it is actually sending/while the client is receiving, and it trigger this bug.
After upgrading from httpx 0.18.2 to 0.21.1 we also suffer from this problem again. As a workaround we pinned httpx to the old version where everything seems to work fine.
fyi @marns93
@tomchristie so can this please be reopened? We can reproduce this again with thousands of connections against HTTP/2 server - Undertow.
Can you make sure you've got the latest version of httpcore
installed?
Hi, I have a similar issue with FastAPI and uvicorn. Python 3.8.
It's probably not the best place to post it, I post it here because it is thanks to this thread that I was able to create a toy program that reproduces the bug. Any help would be appreciated.
Thore are the step to reproduce it:
pip install fastapi
pip install uvicorn
server script (from @yeraydiazdiaz )
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
@app.post("/")
def index():
return {"ok": 1, "data": "welcome to test app 11111111111!"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8001)
On the client:
$ curl --request POST --url http://localhost:8001/
I have this Error that happens sometimes. It often happens multiple times in a row and after 3 to 10 queries it starts working.
I think it happens more ofter when the client is remote (through a VPN) but it also happens with the client on localhost.
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 129, in handle_events
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_connection.py", line 443, in next_event
because the peer has finished their part of the current
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
# in-place modification preserved it and we can just re-raise:
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_connection.py", line 425, in next_event
"""Parse the next event out of our receive buffer, update our internal
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
other failures to read are indicated using other mechanisms
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_readers.py", line 72, in maybe_read_from_IDLE_client
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_util.py", line 88, in validate
if not match:
h11._util.RemoteProtocolError: illegal request line: bytearray(b'HTTP/1.1 302 Found')
(bug_http-V1yNRNQW) lsamper@servername:~/bug_http$ pip freeze
anyio==3.6.1
asgiref==3.5.1
click==8.1.3
fastapi==0.78.0
h11==0.13.0
idna==3.3
pydantic==1.9.0
sniffio==1.2.0
starlette==0.19.1
typing-extensions==4.2.0
uvicorn==0.17.6
For me, updating my libraries fixed the issue : )
I can reproduce this with uvicorn+fastapi and just two requests (no need for thousands). First request failed due to the remote, second request failed do due this issue.
I intermittently got this error when load testing uvicorn endpoint.
This error comes from a proxy endpoint where I am also using
encode/http3
to perform HTTP client calls.