search

encode / httpx

A next generation HTTP client for Python. 🦋
https://www.python-httpx.org/
BSD 3-Clause "New" or "Revised" License
12.78k stars 813 forks source link

h11._util.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE #96

Closed didip closed 3 years ago

didip commented 5 years ago

I intermittently got this error when load testing uvicorn endpoint.

This error comes from a proxy endpoint where I am also using encode/http3 to perform HTTP client calls.

  File "/project/venv/lib/python3.7/site-packages/http3/client.py", line 365, in post
    timeout=timeout,
  File "/project/venv/lib/python3.7/site-packages/http3/client.py", line 497, in request
    timeout=timeout,
  File "/project/venv/lib/python3.7/site-packages/http3/client.py", line 112, in send
    allow_redirects=allow_redirects,
  File "/project/venv/lib/python3.7/site-packages/http3/client.py", line 145, in send_handling_redirects
    request, verify=verify, cert=cert, timeout=timeout
  File "/project/venv/lib/python3.7/site-packages/http3/dispatch/connection_pool.py", line 121, in send
    raise exc
  File "/project/venv/lib/python3.7/site-packages/http3/dispatch/connection_pool.py", line 116, in send
    request, verify=verify, cert=cert, timeout=timeout
  File "/project/venv/lib/python3.7/site-packages/http3/dispatch/connection.py", line 59, in send
    response = await self.h11_connection.send(request, timeout=timeout)
  File "/project/venv/lib/python3.7/site-packages/http3/dispatch/http11.py", line 65, in send
    event = await self._receive_event(timeout)
  File "/project/venv/lib/python3.7/site-packages/http3/dispatch/http11.py", line 109, in _receive_event
    event = self.h11_state.next_event()
  File "/project/venv/lib/python3.7/site-packages/h11/_connection.py", line 439, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/project/venv/lib/python3.7/site-packages/h11/_util.py", line 72, in _reraise_as_remote_protocol_error
    raise self
  File "/project/venv/lib/python3.7/site-packages/h11/_connection.py", line 422, in next_event
    self._process_event(self.their_role, event)
  File "/project/venv/lib/python3.7/site-packages/h11/_connection.py", line 238, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "/project/venv/lib/python3.7/site-packages/h11/_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "/project/venv/lib/python3.7/site-packages/h11/_state.py", line 253, in _fire_event_triggered_transitions
    .format(event_type.__name__, role, self.states[role]))
h11._util.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
vyahello commented 4 years ago

Hey everyone, do we have some progress on this? I see the same error after delayed calls e.g:

import time
from httpx import Client

client = Client()
client.get("/")
time.sleep(300)
client.get("/")
.....
h11._util.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

Does anyone know some workarounds for this? I'm just wondering why I can't see such error in requests module ..

vyahello commented 4 years ago

Hey, will changes be introduced in version above 0.7.8, right?

florimondmanca commented 4 years ago

@vyahello Yes, this fix along with some others will be released soon, probably as 0.7.9. :) If you need them now you can install from master.

vyahello commented 4 years ago

thanks!

andreportela commented 4 years ago

Hi guys.

I am getting a similar exception running my FastAPI app like 

gunicorn -k uvicorn.workers.UvicornH11Worker
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 338, in timeout_keep_alive_handler
    self.conn.send(event)
  File "/usr/local/lib/python3.7/site-packages/h11/_connection.py", line 464, in send
    data_list = self.send_with_data_passthrough(event)
  File "/usr/local/lib/python3.7/site-packages/h11/_connection.py", line 490, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "/usr/local/lib/python3.7/site-packages/h11/_connection.py", line 238, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "/usr/local/lib/python3.7/site-packages/h11/_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "/usr/local/lib/python3.7/site-packages/h11/_state.py", line 253, in _fire_event_triggered_transitions
    .format(event_type.__name__, role, self.states[role]))
h11._util.LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_BODY
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 383, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 139, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 134, in __call__
    await self.error_middleware(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 178, in __call__
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 156, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__
    await response(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 200, in __call__
    await send({"type": "http.response.body", "body": b"", "more_body": False})
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 153, in _send
    await send(message)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 477, in send
    output = self.conn.send(event)
  File "/usr/local/lib/python3.7/site-packages/h11/_connection.py", line 464, in send
    data_list = self.send_with_data_passthrough(event)
  File "/usr/local/lib/python3.7/site-packages/h11/_connection.py", line 480, in send_with_data_passthrough
    "Can't send data when our state is ERROR")
h11._util.LocalProtocolError: Can't send data when our state is ERROR

python 3.7

fastapi==0.42.0
gunicorn==19.9.0
h11==0.8.1
uvicorn==0.10.0

I get it intermittently in a somewhat low but significant rate on my API. Like 8 errors in a 1000 requests sample.

Do you think is somehow related?

florimondmanca commented 4 years ago

Do you think is somehow related?

Possibly the same situation, but that error is coming from Uvicorn’s usage of h11, not HTTPX’s.

andreportela commented 4 years ago

Possibly the same situation, but that error is coming from Uvicorn’s usage of h11, not HTTPX’s.

Edit: So should I report it on uvicorn's or h11's repo?

uriva commented 4 years ago

Hey, still getting this exception:

httpcore._exceptions.ProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
   raise to_exc(exc) from None
 File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
   self.gen.throw(type, value, traceback)
 File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
   event = self.h11_state.next_event()
 File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 142, in _receive_event
   event = await self._receive_event(timeout)
 File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 115, in _receive_response
   ) = await self._receive_response(timeout)
 File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 62, in request
   return await self.connection.request(method, url, headers, stream, timeout)
 File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 78, in request
   response = await connection.request(
 File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 152, in request
   ) = await transport.request(
 File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1264, in send_single_request
   response = await self.send_single_request(request, timeout)
 File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1232, in send_handling_auth
   response = await self.send_handling_auth(
 File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1195, in send_handling_redirects
   response = await self.send_handling_redirects(
 File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1168, in send
   response = await self.send(
 File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1147, in request
   return await self.request(
 File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1305, in get
   return await client.get(url, timeout=timeout)
 File …
dmig-alarstudios commented 4 years ago

Just found out that it's uvicorn causing this exception, probably some bug. Switching service requested via httpx to hypercorn solved the issue for me.

tomchristie commented 4 years ago

@uriva - Can you provide more information on how we can reproduce that error?

uriva commented 4 years ago

I don't have a reproduction yet. One idea is that this happens when spending too much time doing sync work, while processing an async post request. Does that make any sense?

dmig-alarstudios commented 4 years ago

My case is:

  1. FastAPI services A & B are running under uvicorn
  2. service A opens the connection to service B (async with ServiceBClient() as client: ..., ServiceBClient inherits httpx.AsyncClient)
  3. service A makes 2 subsequent requests to service B
  4. during each request, service B makes a request to the outer world (facebook or apple API)
  5. during the second A->B request ProtocolError is raised in service A with very high probability

If I change 1. to: FastAPI service A is running under uvicorn and FastAPI service B under hypercorn -- my problem is solved

dmig-alarstudios commented 4 years ago

One of the traces to clarify a bit.

│Jul 21 07:21:44 ⋮ ⋮: service_a ERROR:    Exception in ASGI application                                                                                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a Traceback (most recent call last):                                                                                                                                                                     │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 386, in run_asgi                                                                      │
│Jul 21 07:21:44 ⋮ ⋮: service_a     result = await app(self.scope, self.receive, self.send)                                                                                                                                            │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__                                                                            │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await self.app(scope, receive, send)                                                                                                                                                        │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/fastapi/applications.py", line 181, in __call__                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await super().__call__(scope, receive, send)                                                                                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/applications.py", line 102, in __call__                                                                                     │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await self.middleware_stack(scope, receive, send)                                                                                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a     raise exc from None                                                                                                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await self.app(scope, receive, _send)                                                                                                                                                              │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/sentry_sdk/integrations/asgi.py", line 106, in _run_asgi3                                                                             │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await self._run_app(scope, lambda: self.app(scope, receive, send))                                                                                                                          │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/sentry_sdk/integrations/asgi.py", line 143, in _run_app                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a     raise exc from None                                                                                                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/sentry_sdk/integrations/asgi.py", line 140, in _run_app                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await callback()                                                                                                                                                                            │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__                                                                                   │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await self.dispatch_func(request, self.call_next)                                                                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File "./service_a/lib/middleware/filter_session_cookie.py", line 32, in web_front_auth_cookie                                                                                             │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response: Response = await call_next(request)                                                                                                                                                      │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 45, in call_next                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     task.result()                                                                                                                                                                                      │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 38, in coro                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await self.app(scope, receive, send)                                                                                                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__                                                                                   │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await self.dispatch_func(request, self.call_next)                                                                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File "./lib/middleware/correlation_id.py", line 22, in add_request_id                                                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await call_next(request)                                                                                                                                                                    │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 45, in call_next                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     task.result()                                                                                                                                                                                      │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 38, in coro                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await self.app(scope, receive, send)                                                                                                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__                                                                                   │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await self.dispatch_func(request, self.call_next)                                                                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette_context/middleware.py", line 47, in dispatch                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await call_next(request)                                                                                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 45, in call_next                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     task.result()                                                                                                                                                                                      │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/middleware/base.py", line 38, in coro                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await self.app(scope, receive, send)                                                                                                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__                                                                                        │
│Jul 21 07:21:44 ⋮ ⋮: service_a     raise exc from None                                                                                                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__                                                                                        │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await self.app(scope, receive, sender)                                                                                                                                                             │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/routing.py", line 550, in __call__                                                                                          │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await route.handle(scope, receive, send)                                                                                                                                                           │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/routing.py", line 227, in handle                                                                                            │
│Jul 21 07:21:44 ⋮ ⋮: service_a     await self.app(scope, receive, send)                                                                                                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/starlette/routing.py", line 41, in app                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await func(request)                                                                                                                                                                     │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/fastapi/routing.py", line 196, in app                                                                                                 │
│Jul 21 07:21:44 ⋮ ⋮: service_a     raw_response = await run_endpoint_function(                                                                                                                                                        │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/fastapi/routing.py", line 147, in run_endpoint_function                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await dependant.call(**values)                                                                                                                                                              │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File "./service_a/routers/auth/oauth.py", line 235, in apple_callback                                                                                                                            │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await callback(request, OAuthProviders.apple, state, code, error,                                                                                                                           │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File "./service_a/routers/auth/oauth.py", line 151, in callback                                                                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     user = await client.oauth_login_by_uid(                                                                                                                                                            │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File "./service_a/lib/clients/user.py", line 170, in oauth_login_by_uid                                                                                                                          │
│Jul 21 07:21:44 ⋮ ⋮: service_a     resp: 'httpx.Response' = await self.put(                                                                                                                                                           │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpx/_client.py", line 1402, in put                                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await self.request(                                                                                                                                                                         │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpx/_client.py", line 1147, in request                                                                                              │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await self.send(                                                                                                                                                                        │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpx/_client.py", line 1168, in send                                                                                                 │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await self.send_handling_redirects(                                                                                                                                                     │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpx/_client.py", line 1195, in send_handling_redirects                                                                              │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await self.send_handling_auth(                                                                                                                                                          │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpx/_client.py", line 1232, in send_handling_auth                                                                                   │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await self.send_single_request(request, timeout)                                                                                                                                        ├
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpx/_client.py", line 1264, in send_single_request                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     ) = await transport.request(                                                                                                                                                                       ├
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 152, in request                                                                             │
│Jul 21 07:21:44 ⋮ ⋮: service_a     response = await connection.request(                                                                                                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_async/connection.py", line 78, in request                                                                                   │
│Jul 21 07:21:44 ⋮ ⋮: service_a     return await self.connection.request(method, url, headers, stream, timeout)                                                                                                                        │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_async/http11.py", line 62, in request                                                                                       │
│Jul 21 07:21:44 ⋮ ⋮: service_a     ) = await self._receive_response(timeout)                                                                                                                                                          │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_async/http11.py", line 115, in _receive_response                                                                            │
│Jul 21 07:21:44 ⋮ ⋮: service_a     event = await self._receive_event(timeout)                                                                                                                                                         │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_async/http11.py", line 142, in _receive_event                                                                               │
│Jul 21 07:21:44 ⋮ ⋮: service_a     event = self.h11_state.next_event()                                                                                                                                                                │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File "/data/marty-pyenv/services/versions/3.8.1/lib/python3.8/contextlib.py", line 131, in __exit__                                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     self.gen.throw(type, value, traceback)                                                                                                                                                             │
│Jul 21 07:21:44 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions                                                                                  │
│Jul 21 07:21:44 ⋮ ⋮: service_a     raise to_exc(exc) from None                                                                                                                                                                        │
│Jul 21 07:21:44 ⋮ ⋮: service_a httpcore._exceptions.ProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
dmig-alarstudios commented 4 years ago

sometimes the top of stack differs a little bit:


│Jul 21 15:37:27 ⋮ ⋮: service_a     event = await self._receive_event(timeout)                                                                          │
│Jul 21 15:37:27 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_async/http11.py", line 145, in _receive_e    │
│Jul 21 15:37:27 ⋮ ⋮: service_a     data = await self.socket.read(self.READ_NUM_BYTES, timeout)                                                         │
│Jul 21 15:37:27 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_backends/asyncio.py", line 134, in read      │
│Jul 21 15:37:27 ⋮ ⋮: service_a     return await asyncio.wait_for(                                                                                      │
│Jul 21 15:37:27 ⋮ ⋮: service_a   File "/data/marty-pyenv/services/versions/3.8.1/lib/python3.8/contextlib.py", line 131, in __exit__                   │
│Jul 21 15:37:27 ⋮ ⋮: service_a     self.gen.throw(type, value, traceback)                                                                              │
│Jul 21 15:37:27 ⋮ ⋮: service_a   File ".../lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exception    │
│Jul 21 15:37:27 ⋮ ⋮: service_a     raise to_exc(exc) from None                                                                                         │
│Jul 21 15:37:27 ⋮ ⋮: service_a httpcore._exceptions.ReadTimeout                                                                                        ```
uriva commented 4 years ago

@tomchristie should we reopen to make it easier for others to find this issue?

florimondmanca commented 4 years ago

At this point I'd be speculating this is a bug/edge case in Uvicorn, rather than HTTPX behaving badly. Uvicorn might be doing something that causes it to close the connection too early while it is sending data to the client.

What would really help now I think is a detailed and complete reproduction setup with complete code listings, eg based on the description above. If we can strip FastAPI out of the equation (eg use a raw ASGI app), then even better.

We can reopen to mark this as ongoing discussion but once again it's not clear to me yet whether this qualifies as an HTTPX bug.

dmig commented 4 years ago

I can definitely tell one thing: a testcase with 2 simple services running locally doesn't reproduce this case. There is something else that don't get right now: maybe a network delay, a proxy to the outer world (yes, there is one)

dmig-alarstudios commented 4 years ago

Oh! Forgot to mention: today I made tests with --timeout-keep-alive 30. Well... there is a difference:

Hope this helps...

florimondmanca commented 4 years ago

@dmig Could you share the full code you used to try and replicate locally?

dmig-alarstudios commented 4 years ago

Sure!

service_a.py

import logging
import httpx
from fastapi import FastAPI, Response

app = FastAPI()

logging.getLogger().__name__ = __name__

@app.get('/', response_class=Response, status_code=204)
async def main():
    async with httpx.AsyncClient(trust_env=False, base_url='http://localhost:8001', timeout=10) as client:
        logging.info('sending request to app2/ep1')
        resp = await client.get('/ep1')
        logging.info('ep1 resp: %d %s', resp.status_code, resp.reason_phrase)

        logging.info('sending request to app2/ep2')
        resp = await client.get('/ep2')
        logging.info('ep2 resp: %d %s', resp.status_code, resp.reason_phrase)

service_b.py

import logging
import httpx
from fastapi import FastAPI, Response

app = FastAPI(default_response_class=Response)

logging.getLogger().__name__ = __name__

@app.get('/ep1', status_code=204)
async def ep1():
    logging.info('sending request to httpbin')
    async with httpx.AsyncClient(trust_env=False, timeout=10) as client:
        resp = await client.get('https://httpbin.org/delay/6')

    logging.info('resp: %d %s', resp.status_code, resp.reason_phrase)

@app.get('/ep2', status_code=204)
async def ep2():
    logging.info('sending request to httpbin')
    async with httpx.AsyncClient(timeout=10) as client:
        resp = await client.get('https://httpbin.org/delay/6')

    logging.info('resp: %d %s', resp.status_code, resp.reason_phrase)

Basically this is the same code as one that causes ProtocolError/ReadTimeout, but it works as it should.

dmig-alarstudios commented 4 years ago

Should we move the discussion to uvicorn issues?

dmig-alarstudios commented 4 years ago

One more thing that might give a clue: test servers where the error is reproducible are running ancient ubuntu 14.04 with 3.13.0-170-generic kernel

uriva commented 4 years ago

My exception didn't happen within a uvicorn context.

tomchristie commented 3 years ago

I think this will have been resolved by encode/httpcore#112 - once that's released I think you'll find that you're getting a much more clear ReadError raised here, and see that the connection has been closed.

Might also be that asyncio isn't doing a good job of detecting when the connection has been closed by the remote end.

dmig-alarstudios commented 3 years ago

I'm currently able to reproduce a similar (maybe same) problem locally on 0.13.3, it's always ReadTimeout when trying to send second request data:

... skip ...
INFO:     uvicorn.access: 127.0.0.1:53976 - "POST /ep1/ HTTP/1.1" 201 Created
DEBUG [2020-07-30 17:23:57] httpx._client - HTTP Request: POST http://127.0.0.1:9093/ep1/ "HTTP/1.1 201 Created"
DEBUG:    httpx._client: HTTP Request: POST http://127.0.0.1:9093/ep1/ "HTTP/1.1 201 Created"
TRACE [2020-07-30 17:23:57] httpcore._async.http11 - receive_event=Data(<840 bytes>)
TRACE:    httpcore._async.http11: receive_event=Data(<840 bytes>)
TRACE [2020-07-30 17:23:57] httpcore._async.http11 - receive_event=EndOfMessage(headers=[])
TRACE:    httpcore._async.http11: receive_event=EndOfMessage(headers=[])
TRACE [2020-07-30 17:23:57] httpcore._async.http11 - response_closed our_state=DONE their_state=DONE
TRACE:    httpcore._async.http11: response_closed our_state=DONE their_state=DONE
TRACE [2020-07-30 17:23:57] httpcore._async.connection_pool - get_connection_from_pool=(b'http', b'127.0.0.1', 9093)
TRACE:    httpcore._async.connection_pool: get_connection_from_pool=(b'http', b'127.0.0.1', 9093)
TRACE [2020-07-30 17:23:57] httpcore._async.connection_pool - reuse connection=<httpcore._async.connection.AsyncHTTPConnection object at 0x7f5e85807730>
TRACE:    httpcore._async.connection_pool: reuse connection=<httpcore._async.connection.AsyncHTTPConnection object at 0x7f5e85807730>
TRACE [2020-07-30 17:23:57] httpcore._async.connection - connection.request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'x-correlation-id', b'e5065f60-1c2e-4820-9e58-268ac492b8aa'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE:    httpcore._async.connection: connection.request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'x-correlation-id', b'e5065f60-1c2e-4820-9e58-268ac492b8aa'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE [2020-07-30 17:23:57] httpcore._async.http11 - send_request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'x-correlation-id', b'e5065f60-1c2e-4820-9e58-268ac492b8aa'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE:    httpcore._async.http11: send_request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'x-correlation-id', b'e5065f60-1c2e-4820-9e58-268ac492b8aa'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE [2020-07-30 17:23:57] httpcore._async.http11 - send_data=Data(<127 bytes>)
TRACE:    httpcore._async.http11: send_data=Data(<127 bytes>)
TRACE [2020-07-30 17:24:02] httpcore._async.connection_pool - remove from pool connection=<httpcore._async.connection.AsyncHTTPConnection object at 0x7f5e85807730>
TRACE:    httpcore._async.connection_pool: remove from pool connection=<httpcore._async.connection.AsyncHTTPConnection object at 0x7f5e85807730>
INFO:     uvicorn.access: 127.0.0.1:33056 - "POST /api/v1/endpoint/ HTTP/1.1" 500 Internal Server Error
ERROR:    uvicorn.error: Exception in ASGI application
... skip loooong stack trace ...
  File "./service_a/lib/mylib.py", line 239, in register
    await client.call_ep2(**params)
  File "./service_a/lib/clients/service_b.py", line 44, in call_ep2
    resp: 'httpx.Response' = await self.patch('/ep2/', json=kwargs)
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 1430, in patch
    return await self.request(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 1147, in request
    response = await self.send(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 1168, in send
    response = await self.send_handling_redirects(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 1195, in send_handling_redirects
    response = await self.send_handling_auth(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 1232, in send_handling_auth
    response = await self.send_single_request(request, timeout)
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 1264, in send_single_request
    ) = await transport.request(
  File ".../lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 152, in request
    response = await connection.request(
  File ".../lib/python3.8/site-packages/httpcore/_async/connection.py", line 78, in request
    return await self.connection.request(method, url, headers, stream, timeout)
  File ".../lib/python3.8/site-packages/httpcore/_async/http11.py", line 62, in request
    ) = await self._receive_response(timeout)
  File ".../lib/python3.8/site-packages/httpcore/_async/http11.py", line 115, in _receive_response
    event = await self._receive_event(timeout)
  File ".../lib/python3.8/site-packages/httpcore/_async/http11.py", line 145, in _receive_event
    data = await self.socket.read(self.READ_NUM_BYTES, timeout)
  File ".../lib/python3.8/site-packages/httpcore/_backends/asyncio.py", line 134, in read
    return await asyncio.wait_for(
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File ".../lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore._exceptions.ReadTimeout

Hope it helps.

Tomorrow I'll try to debug this.

tomchristie commented 3 years ago

The best thing to do next would be to find out if it replicates with the sync Client, or with the AsyncClient while using trio.

Also if there's any way we can replicate this locally that'd be amazing, eg does it replicate consistently against some particular external url? Does it replicate without proxies or only with them?

dmig-alarstudios commented 3 years ago

It does reproduce consistently, all communication is between 2 local services. No proxies, no SSL. But I still don't have enough ifno to create a small separate testcase: it is easily reproducible with one endpoint, but never with another...

dmig-alarstudios commented 3 years ago

100% reproducible with sync Client:

... skip ...
INFO:     uvicorn.access: 127.0.0.1:56226 - "POST /ep1/ HTTP/1.1" 201 Created
DEBUG [2020-07-31 10:10:31] httpx._client - HTTP Request: POST http://127.0.0.1:9093/ep1/ "HTTP/1.1 201 Created"
DEBUG:    httpx._client: HTTP Request: POST http://127.0.0.1:9093/ep1/ "HTTP/1.1 201 Created"
TRACE [2020-07-31 10:10:31] httpcore._sync.http11 - receive_event=Data(<840 bytes>)
TRACE:    httpcore._sync.http11: receive_event=Data(<840 bytes>)
TRACE [2020-07-31 10:10:31] httpcore._sync.http11 - receive_event=EndOfMessage(headers=[])
TRACE:    httpcore._sync.http11: receive_event=EndOfMessage(headers=[])
TRACE [2020-07-31 10:10:31] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
TRACE:    httpcore._sync.http11: response_closed our_state=DONE their_state=DONE
TRACE [2020-07-31 10:10:31] httpcore._sync.connection_pool - get_connection_from_pool=(b'http', b'127.0.0.1', 9093)
TRACE:    httpcore._sync.connection_pool: get_connection_from_pool=(b'http', b'127.0.0.1', 9093)
TRACE [2020-07-31 10:10:31] httpcore._sync.connection_pool - reuse connection=<httpcore._sync.connection.SyncHTTPConnection object at 0x7fde344640d0>
TRACE:    httpcore._sync.connection_pool: reuse connection=<httpcore._sync.connection.SyncHTTPConnection object at 0x7fde344640d0>
TRACE [2020-07-31 10:10:31] httpcore._sync.connection - connection.request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE:    httpcore._sync.connection: connection.request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE [2020-07-31 10:10:31] httpcore._sync.http11 - send_request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE:    httpcore._sync.http11: send_request method=b'PATCH' url=(b'http', b'127.0.0.1', 9093, b'/ep2/') headers=[(b'host', b'127.0.0.1:9093'), (b'user-agent', b'python-httpx/0.13.3'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'content-length', b'127'), (b'content-type', b'application/json')]
TRACE [2020-07-31 10:10:31] httpcore._sync.http11 - send_data=Data(<127 bytes>)
TRACE:    httpcore._sync.http11: send_data=Data(<127 bytes>)
TRACE [2020-07-31 10:10:36] httpcore._sync.connection_pool - remove from pool connection=<httpcore._sync.connection.SyncHTTPConnection object at 0x7fde344640d0>
TRACE:    httpcore._sync.connection_pool: remove from pool connection=<httpcore._sync.connection.SyncHTTPConnection object at 0x7fde344640d0>
INFO:     uvicorn.access: 127.0.0.1:35306 - "POST /api/v1/endpoint/ HTTP/1.1" 500 Internal Server Error
ERROR:    uvicorn.error: Exception in ASGI application
Traceback (most recent call last):

... skip ...

  File "./service_a/lib/mylib.py", line 249, in register
    resp = client.patch('/ep2/',
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 880, in patch
    return self.request(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 600, in request
    return self.send(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 620, in send
    response = self.send_handling_redirects(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 647, in send_handling_redirects
    response = self.send_handling_auth(
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 684, in send_handling_auth
    response = self.send_single_request(request, timeout)
  File ".../lib/python3.8/site-packages/httpx/_client.py", line 714, in send_single_request
    ) = transport.request(
  File ".../lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 152, in request
    response = connection.request(
  File ".../lib/python3.8/site-packages/httpcore/_sync/connection.py", line 78, in request
    return self.connection.request(method, url, headers, stream, timeout)
  File ".../lib/python3.8/site-packages/httpcore/_sync/http11.py", line 62, in request
    ) = self._receive_response(timeout)
  File ".../lib/python3.8/site-packages/httpcore/_sync/http11.py", line 115, in _receive_response
    event = self._receive_event(timeout)
  File ".../lib/python3.8/site-packages/httpcore/_sync/http11.py", line 145, in _receive_event
    data = self.socket.read(self.READ_NUM_BYTES, timeout)
  File ".../lib/python3.8/site-packages/httpcore/_backends/sync.py", line 62, in read
    return self.sock.recv(n)
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File ".../lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore._exceptions.ReadTimeout: timed out
dmig-alarstudios commented 3 years ago

sync code is:

    with httpx.Client(base_url='http://127.0.0.1:9093') as client:
        resp: httpx.Response = client.post(
            '/ep1/',
            data=orjson.dumps(somedata),
            headers={'Content-Type': 'application/json', 'X-Application': app_id}
        )
        forward_response_error(resp)

        resp_data = orjson.loads(resp.content)

        resp = client.patch('/ep2/', json=someotherdata)
        forward_response_error(resp)

async:

    async with httpx.AsyncClient(base_url='http://127.0.0.1:9093') as client:
        resp: httpx.Response = await client.post(
            '/ep1/',
            data=orjson.dumps(somedata),
            headers={'Content-Type': 'application/json', 'X-Application': app_id}
        )
        forward_response_error(resp)

        resp_data = orjson.loads(resp.content)

        resp = await client.patch('/ep2/', json=someotherdata)
        forward_response_error(resp)

reconnect (one more async with httpx.AsyncClient(base_url='http://127.0.0.1:9093') as client:) before the second request solves the problem

florimondmanca commented 3 years ago

@dmig-alarstudios Thanks for the sync code snippet. However the Client() is missing a timeout=10, so assuming you're still using the service that calls the /delay/6 endpoint on HTTPBin, the ReadTimeout is to be expected.

florimondmanca commented 3 years ago

I tried to reproduce this myself based on https://github.com/encode/httpx/issues/96#issuecomment-662290816, and wasn't able to.

@dmig-alarstudios Can you try the code below and let me know if it works fine on your side?

Web app that proxies requests to HTTPBin:

# app.py
import httpx
from starlette.applications import Starlette
from starlette.routing import Route
from starlette.responses import PlainTextResponse

async def ep1(request):
    async with httpx.AsyncClient(timeout=10) as client:
        print("ep1 request")
        await client.get("https://httpbin.org/delay/6")

    return PlainTextResponse("OK")

async def ep2(request):
    async with httpx.AsyncClient(timeout=10) as client:
        print("ep2 request")
        await client.get("https://httpbin.org/delay/6")

    return PlainTextResponse("OK")

app = Starlette(routes=[Route("/ep1", ep1), Route("/ep2", ep2)])

Client script (I assume this doesn't need to be wrapped in a web app?):

# client.py
import httpx

async def main():
    async with httpx.AsyncClient(timeout=10) as client:
        print("ep1")
        await client.get("http://localhost:8000/ep1")
        print("ep2")
        await client.get("http://localhost:8000/ep2")

if __name__ == "__main__":
    import asyncio
    import trio

    asyncio.run(main())
    trio.run(main)

Start the web app:

HTTPX_LOG_LEVEL=trace uvicorn app:app

Run the client script:

HTTPX_LOG_LEVEL=trace python client.py
dmig-alarstudios commented 3 years ago

I tried to reproduce this myself based on #96 (comment), and wasn't able to.

I've said that it should reproduce the problem but it doesn't.

dmig-alarstudios commented 3 years ago

However the Client() is missing a timeout=10...

Timeout doesn't matter. If I set it to 10 -- I'll just have to wait 10 seconds before the ReadTimeout -- tried that already. Have a close look at trace, especially at timings: it tries to send data for the second request but fails.

florimondmanca commented 3 years ago

Okay well, can you then share the full, minimal reproduction code for your example in https://github.com/encode/httpx/issues/96#issuecomment-667002995, since that seems to reproduce the issue correctly?

What we really need here is a full, minimal, dependency-free code listing like I did in https://github.com/encode/httpx/issues/96#issuecomment-667019998. This would allow anyone to reproduce the issue at least somewhat consistently. Then we can go and debug what's wrong. :confused:

dmig-alarstudios commented 3 years ago

Okay well, can you then share the full, minimal reproduction code for your example in #96 (comment), since that seems to reproduce the issue correctly?

Still working on that... :(

There is definitely something related to the first service_b response: maybe something remains unread?

tomchristie commented 3 years ago

Also helpful, what behaviour does requests have here? And to be 100% clear - does this only replicate when sent via a proxy?

dmig-alarstudios commented 3 years ago

@tomchristie my last case -- no proxies, no SSL. I'll make a test using requests.

tomchristie commented 3 years ago

And are you testing against an external URL, or a service that's running locally?

dmig-alarstudios commented 3 years ago

No: https://github.com/encode/httpx/issues/96#issuecomment-666976890 2 services locally, interacting with each other.

And yes, it's 100% reproducible with requests. Definitely not an httpx issue but uvicorn.

dmig-alarstudios commented 3 years ago 
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:9093
send: b'POST /ep1/ HTTP/1.1\r\nHost: 127.0.0.1:9093\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 214\r\n\r\n'
send: b'some data'
reply: 'HTTP/1.1 201 Created\r\n'
header: date: Fri, 31 Jul 2020 13:53:22 GMT
header: server: uvicorn
header: content-length: 835
header: content-type: application/json
header: x-correlation-id: 23856694-cc1e-46a0-a9dc-b735eaa1ac9c
DEBUG:urllib3.connectionpool:http://127.0.0.1:9093 "POST /ep1/ HTTP/1.1" 201 835

send: b'PATCH /ep2/ HTTP/1.1\r\nHost: 127.0.0.1:9093\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 122\r\nContent-Type: application/json\r\n\r\n'
send: b'some other data'
reply: ''
Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File ".../lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File ".../lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test-with-requests.py", line 39, in <module>
    resp = client.patch(base_url + '/ep2/',
  File ".../lib/python3.8/site-packages/requests/sessions.py", line 602, in patch
    return self.request('PATCH', url, data=data, **kwargs)
  File ".../lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File ".../lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File ".../lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

requests raises exactly same RemoteDisconnect exception. it's strange, that httpx changed behavior.

tomchristie commented 3 years ago

Well that narrows it down to either:

  1. An issue with your application, itself. It's just never returning a response.
  2. An genuine issue with uvicorn.

You could narrow that down for yourself by running the same application with hypercorn and/or daphne and seeing what behaviour it exhibits then.

tomchristie commented 3 years ago

In any case, no, not an httpx issue, but we could have some clearer messaging in our exceptions here.

There ought to be an improvement with the upcoming 0.14, since we properly identify closed connections vs raising a protocol error from h11 if a socket read operation returns b""

dmig-alarstudios commented 3 years ago

I'm pretty sure, that's a uvicorn issue. Because running my service under hypercorn solves this problem:

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:9093
send: b'POST /ep1/ HTTP/1.1\r\nHost: 127.0.0.1:9093\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 214\r\n\r\n'
send: b'some data'
reply: 'HTTP/1.1 201 \r\n'
header: content-length: 835
header: content-type: application/json
header: x-correlation-id: d4ade3c6-e50f-4a6b-9e25-13a1d609fab5
header: date: Fri, 31 Jul 2020 14:11:57 GMT
header: server: hypercorn-h11
DEBUG:urllib3.connectionpool:http://127.0.0.1:9093 "POST /ep1/ HTTP/1.1" 201 835

send: b'PATCH /ep2/ HTTP/1.1\r\nHost: 127.0.0.1:9093\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 122\r\nContent-Type: application/json\r\n\r\n'
send: b'some other data'
reply: 'HTTP/1.1 204 \r\n'
header: x-correlation-id: 9f653cf6-7cc8-4df1-903c-0fbe0d5f2693
header: date: Fri, 31 Jul 2020 14:11:57 GMT
header: server: hypercorn-h11
DEBUG:urllib3.connectionpool:http://127.0.0.1:9093 "PATCH /ep2/ HTTP/1.1" 204 0
tomchristie commented 3 years ago

Something we might want to do... if we get a ReadTimeout, check .is_connection_dropped and include that information in the exception text.

dmig-alarstudios commented 3 years ago

@tomchristie since I have an environment with 100% reproducible issue, can you point me where to look at, to debug the issue in uvicorn?

tomchristie commented 3 years ago

Start by checking it run against hypercorn. If it works on hypercorn but not uvicorn, then file an issue on the uvicorn repo.

dmig-alarstudios commented 3 years ago

Ok, nailed it: https://github.com/encode/starlette/issues/919

elupus commented 3 years ago

Is this issue resolved from a client perspective? We seem to be getting a similar error with a rest client. I don't have a backtrace yet though not a good way to reproduce.

lmmx commented 3 years ago

Not sure if this is "won't fix" now, but I'm getting the same error message which seems to stem from event = self.h11_state.next_event() and leads to httpx.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE as documented above.

Not using uvicorn, traceback below

Click to show traceback

``` Traceback (most recent call last): File "/home/louis/miniconda3/lib/python3.8/site-packages/httpx/_exceptions.py", line 326, in map_exceptions yield File "/home/louis/miniconda3/lib/python3.8/site-packages/httpx/_client.py", line 1492, in _send_single_request (status_code, headers, stream, ext) = await transport.arequest( File "/home/louis/miniconda3/lib/python3.8/site-packages/httpx/_transports/default.py", line 169, in arequest return await self._pool.arequest( File "/home/louis/miniconda3/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 218, in arequest response = await connection.arequest( File "/home/louis/miniconda3/lib/python3.8/site-packages/httpcore/_async/connection.py", line 106, in arequest return await self.connection.arequest(method, url, headers, stream, ext) File "/home/louis/miniconda3/lib/python3.8/site-packages/httpcore/_async/http11.py", line 72, in arequest ) = await self._receive_response(timeout) File "/home/louis/miniconda3/lib/python3.8/site-packages/httpcore/_async/http11.py", line 133, in _receive_response event = await self._receive_event(timeout) File "/home/louis/miniconda3/lib/python3.8/site-packages/httpcore/_async/http11.py", line 169, in _receive_event event = self.h11_state.next_event() File "/home/louis/miniconda3/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/louis/miniconda3/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions raise to_exc(exc) from None httpcore.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "/home/louis/dev/beeb/src/beeb/nav/sched/catalogue.py", line 17, in __init__ self.async_pull_and_parse(listings) File "/home/louis/dev/beeb/src/beeb/nav/sched/catalogue.py", line 46, in async_pull_and_parse fetch_episode_metadata(listings, pbar=pbar, verbose=verbose) File "/home/louis/dev/beeb/src/beeb/nav/sched/async_utils.py", line 69, in fetch_episode_metadata return asyncio.run(async_fetch_episodes(listings, pbar, verbose)) File "/home/louis/miniconda3/lib/python3.8/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/home/louis/miniconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/home/louis/dev/beeb/src/beeb/nav/sched/async_utils.py", line 65, in async_fetch_episodes return await zs File "/home/louis/miniconda3/lib/python3.8/site-packages/aiostream/core.py", line 32, in wait_stream ```

A bit of a 'Heisenbug', happens less than 10% of the time when requesting (many) JSON files for episode metadata from the BBC's radio schedules. Using my library beeb I get it from beeb.nav.sched.ProgrammeCatalogue("r4", n_days=30) and can greatly increase its frequency by switching my httpx.AsyncClient to use http2=True.

Not disastrous, I will probably just catch and retry, but thought I should pitch in with my experience :smiley:

dmig-alarstudios commented 3 years ago

@lmmx look at https://github.com/encode/httpx/issues/96#issuecomment-672884672 -- there is a workaround