Closed didip closed 4 years ago
tomchristie
commented
3 years ago A bit of a 'Heisenbug', happens less than 10% of the time when requesting (many) JSON files for episode metadata from the BBC's radio schedules.
@lmmx - Could you provide enough information to reproduce this? I'd be happy to spend some time looking into it.
dmig-alarstudios
commented
3 years ago also answering 'yes' to any of these questions means that this is not a httpx problem:
BackgroundTasks?BaseHTTPMiddleware or middleware decorator from Starlette?
lioncui
commented
2 years ago i have same problem with fastapi + uvicorn. can i use uwsgi replace uvicorn to fix it ?
jouve
commented
2 years ago I could reproduce with this snippet:
import asyncio
async def main():
async with AsyncClient() as session:
await asyncio.wait([
asyncio.create_task(session.get('http://localhost:80'))
for _ in range(1000)
])
asyncio.run(main())the ConnectionClosed seems to be generated only in this place in h11:
def _extract_next_receive_event(self):
state = self.their_state
# We don't pause immediately when they enter DONE, because even in
# DONE state we can still process a ConnectionClosed() event. But
# if we have data in our buffer, then we definitely aren't getting
# a ConnectionClosed() immediately and we need to pause.
if state is DONE and self._receive_buffer:
return PAUSED
if state is MIGHT_SWITCH_PROTOCOL or state is SWITCHED_PROTOCOL:
return PAUSED
assert self._reader is not None
event = self._reader(self._receive_buffer)
if event is None:
if not self._receive_buffer and self._receive_buffer_closed:
# In some unusual cases (basically just HTTP/1.0 bodies), EOF
# triggers an actual protocol event; in that case, we want to
# return that event, and then the state will change and we'll
# get called again to generate the actual ConnectionClosed().
if hasattr(self._reader, "read_eof"):
event = self._reader.read_eof()
else:
event = ConnectionClosed()
if event is None:
event = NEED_DATA
return eventand _receive_buffer_closed seems to be closed only in
def receive_data(self, data):
if data:
if self._receive_buffer_closed:
raise RuntimeError("received close, then received more data?")
self._receive_buffer += data
else:
self._receive_buffer_closed = Truemeaning it was called with an empty data
in turn, this comes from httpcore:
async def _receive_event(self, timeout: float = None) -> H11Event:
while True:
with map_exceptions({h11.RemoteProtocolError: RemoteProtocolError}):
event = self._h11_state.next_event()
if event is h11.NEED_DATA:
data = await self._network_stream.read(
self.READ_NUM_BYTES, timeout=timeout
)
self._h11_state.receive_data(data)
else:
return event
so _network_stream.read returned empty data
as I'm using asyncio, it's an AsyncIOStream with this code:
async def read(self, max_bytes: int, timeout: float = None) -> bytes:
exc_map = {
TimeoutError: ReadTimeout,
anyio.BrokenResourceError: ReadError,
}
with map_exceptions(exc_map):
with anyio.fail_after(timeout):
try:
return await self._stream.receive(max_bytes=max_bytes)
except anyio.EndOfStream: # pragma: nocover
return b""so I guess we reached EndOfStream on the client side then emitted the transition (their_role=SERVER, state=SEND_RESPONSE, event_type=ConnectionClosed) which is not managed.
When programming on the server side, it means client reached ConnectionClosed while the server is the sending the response/client is receiving, I think the bug does not happen since the server would not be calling receive_data while sending the response.
On the client side, it means the server reached ConnectionClosed while it is actually sending/while the client is receiving, and it trigger this bug.
mm-matthias
commented
2 years ago After upgrading from httpx 0.18.2 to 0.21.1 we also suffer from this problem again. As a workaround we pinned httpx to the old version where everything seems to work fine.
fyi @marns93
zyv
commented
2 years ago @tomchristie so can this please be reopened? We can reproduce this again with thousands of connections against HTTP/2 server - Undertow.
tomchristie
commented
2 years ago Can you make sure you've got the latest version of httpcore installed?
lsamper
commented
2 years ago Hi, I have a similar issue with FastAPI and uvicorn. Python 3.8.
It's probably not the best place to post it, I post it here because it is thanks to this thread that I was able to create a toy program that reproduces the bug. Any help would be appreciated.
Thore are the step to reproduce it:
pip install fastapi
pip install uvicornserver script (from @yeraydiazdiaz )
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
@app.post("/")
def index():
return {"ok": 1, "data": "welcome to test app 11111111111!"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8001)On the client:
$ curl --request POST --url http://localhost:8001/I have this Error that happens sometimes. It often happens multiple times in a row and after 3 to 10 queries it starts working.
I think it happens more ofter when the client is remote (through a VPN) but it also happens with the client on localhost.
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 129, in handle_events
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_connection.py", line 443, in next_event
because the peer has finished their part of the current
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
# in-place modification preserved it and we can just re-raise:
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_connection.py", line 425, in next_event
"""Parse the next event out of our receive buffer, update our internal
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
other failures to read are indicated using other mechanisms
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_readers.py", line 72, in maybe_read_from_IDLE_client
File "/home/lsamper/.local/share/virtualenvs/bug_http-V1yNRNQW/lib/python3.8/site-packages/h11/_util.py", line 88, in validate
if not match:
h11._util.RemoteProtocolError: illegal request line: bytearray(b'HTTP/1.1 302 Found')(bug_http-V1yNRNQW) lsamper@servername:~/bug_http$ pip freeze
anyio==3.6.1
asgiref==3.5.1
click==8.1.3
fastapi==0.78.0
h11==0.13.0
idna==3.3
pydantic==1.9.0
sniffio==1.2.0
starlette==0.19.1
typing-extensions==4.2.0
uvicorn==0.17.6
fredi-python
commented
10 months ago For me, updating my libraries fixed the issue : )
lmaddox
commented
3 weeks ago I can reproduce this with uvicorn+fastapi and just two requests (no need for thousands). First request failed due to the remote, second request failed do due this issue.
I intermittently got this error when load testing uvicorn endpoint.
This error comes from a proxy endpoint where I am also using
encode/http3to perform HTTP client calls.