wangluozhe / requests

用于快速请求HTTP或HTTPS,并支持修改ja3、ja4指纹
GNU General Public License v3.0
424 stars 96 forks source link

fastapi服务端解析数据时提示:Content-Length can't be present with Transfer-Encoding #3

Closed economic-tyro closed 2 years ago

economic-tyro commented 2 years ago

错误描述: 我服务端是用python-fastapi架的,完整错误提示是:

WARNING:  Invalid HTTP request received.
Traceback (most recent call last):
  File "D:\pythonEnv\JkEnv\lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 125, in data_received
    self.parser.feed_data(data)
  File "httptools\parser\parser.pyx", line 212, in httptools.parser.parser.HttpParser.feed_data
httptools.parser.errors.HttpParserError: Content-Length can't be present with Transfer-Encoding

采用requests的Post方法发送数据,服务端收到的数据为: b'POST /js HTTP/1.1\r\nAccept: /\r\nAccept-Encoding: gzip, deflate, br\r\nConnection: keep-alive\r\nContent-Length: 17\r\nContent-Type: application/x-www-form-urlencoded\r\nHost: localhost:8086\r\nTransfer-Encoding: chunked\r\nUser-Agent: golang-requests1.0.2\r\n\r\n11\r\nparams=100&Ua=100\r\n0\r\n\r\n'

而对于同样的数据,若采用python-requests的post方法,则能够成功,服务端收到的数据为: b'POST /js HTTP/1.1\r\nHost: 127.0.0.1:8086\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: /\r\nConnection: keep-alive\r\nContent-Length: 17\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\nparams=100&Ua=100'

其他的信息: golang的代码为(记为code-1):

func test_post() {
    data := url.NewData()
    data.Set("params", "100")
    data.Set("Ua", "100")
    r, nil := requests.Post("http://localhost:8086/js", &url.Request{Data: data})
    fmt.Print(nil)
    fmt.Printf(r.Text)
}

fastapi端的相关代码为:


@app.post("/js")
def params(params=Form(None), Ua=Form(None)):
    return {"code": 200, 'params': params, 'ua': Ua}

另外,还进行了一下对比实验: 1.采用golang的http向我架的服务端post数据,即

resp, err := http.Post("http://localhost:8086/js",
        "application/x-www-form-urlencoded",
        strings.NewReader("params=10&Ua=100"))

,结果正常;

2.采用python的requests向我架的服务端post数据,结果正常; 3.采用作者的requests包(code-1)向 http://httpbin.org/post 发包,结果正常。

最后,查阅资料得出的模糊结论

可能是requests发包时产生了Transfer-Encoding: chunked,并进而使fastapi解析错误?

wangluozhe commented 2 years ago

错误描述: 我服务端是用python-fastapi架的,完整错误提示是:

WARNING:  Invalid HTTP request received.
Traceback (most recent call last):
  File "D:\pythonEnv\JkEnv\lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 125, in data_received
    self.parser.feed_data(data)
  File "httptools\parser\parser.pyx", line 212, in httptools.parser.parser.HttpParser.feed_data
httptools.parser.errors.HttpParserError: Content-Length can't be present with Transfer-Encoding

采用requests的Post方法发送数据,服务端收到的数据为: b'POST /js HTTP/1.1\r\nAccept: /\r\nAccept-Encoding: gzip, deflate, br\r\nConnection: keep-alive\r\nContent-Length: 17\r\nContent-Type: application/x-www-form-urlencoded\r\nHost: localhost:8086\r\nTransfer-Encoding: chunked\r\nUser-Agent: golang-requests1.0.2\r\n\r\n11\r\nparams=100&Ua=100\r\n0\r\n\r\n'

而对于同样的数据,若采用python-requests的post方法,则能够成功,服务端收到的数据为: b'POST /js HTTP/1.1\r\nHost: 127.0.0.1:8086\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: /\r\nConnection: keep-alive\r\nContent-Length: 17\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\nparams=100&Ua=100'

其他的信息: golang的代码为(记为code-1):

func test_post() {
  data := url.NewData()
  data.Set("params", "100")
  data.Set("Ua", "100")
  r, nil := requests.Post("http://localhost:8086/js", &url.Request{Data: data})
  fmt.Print(nil)
  fmt.Printf(r.Text)
}

fastapi端的相关代码为:


@app.post("/js")
def params(params=Form(None), Ua=Form(None)):
    return {"code": 200, 'params': params, 'ua': Ua}

另外,还进行了一下对比实验: 1.采用golang的http向我架的服务端post数据,即

resp, err := http.Post("http://localhost:8086/js",
      "application/x-www-form-urlencoded",
      strings.NewReader("params=10&Ua=100"))

,结果正常;

2.采用python的requests向我架的服务端post数据,结果正常; 3.采用作者的requests包(code-1)向 http://httpbin.org/post 发包,结果正常。

最后,查阅资料得出的模糊结论

可能是requests发包时产生了Transfer-Encoding: chunked,并进而使fastapi解析错误?

我这边测试表示没有任何问题,python用的fastapi,代码跟你的一样,golang用的golang-requests,也没有任何问题,python的fastapi正常接收数据

economic-tyro commented 2 years ago

我之前项目是用的python3.8.7,今天把版本升级到3.10.4,在代码不变的情况下进行测试,结果可以了。所以,应该是python端的问题,可能是旧版本对数据解析不符合协议。golang-requests包很好用,辛苦作者了。