search

go-resty / resty

Simple HTTP and REST client library for Go
MIT License
10.02k stars 706 forks source link

How to set Content-Length header #395

Closed vijayrajah closed 3 years ago

vijayrajah commented 3 years ago

I'm trying to call an API, that expects 'Content-Length' = 0 for a HTTP PATCH Request. There is no request body for this request.

It seems golang (from 1.8) does not allow to set Content-length with zero bytes. ( pls see -- https://github.com/golang/go/issues/20257)

How do I set the header? I have tried

restyClient.SetBody(http.NoBody).SetContentLength(true)

And

reqHeader := map[string]string {
        "Content-Length": "0",
}

None of these 2 ways set the content-length header.

moorereason commented 3 years ago

I'm not able to reproduce this issue. Can you provide a reproducible example?

package main

import (
        "fmt"

        "github.com/go-resty/resty/v2"
)

func main() {
        resp, err := resty.New().R().Patch("https://httpbin.org/anything")
        if err != nil {
                panic(err)
        }

        fmt.Printf("%s\n", resp.Body())
}
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Content-Length": "0",
    "Host": "httpbin.org",
    "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)",
    "X-Amzn-Trace-Id": "Root=1-5fce910b-2d8546fb35643ed93aab435b"
  },
  "json": null,
  "method": "PATCH",
  "url": "https://httpbin.org/anything"
}
vijayrajah commented 3 years ago

@moorereason Thanks for looking into this issue

I added debug.. so this is my test code now

func main(){
c1 := createHTTPClient("test-token").R()

    reqFlushQuery := map[string]string {
        "action": "flush",
        "position": "11111",
    }

    c1.SetQueryParams(reqFlushQuery)
    resp, err := c1.Patch("https://httpbin.org/anything")
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s\n", resp.Body())
}

func createHTTPClient(bt string) *resty.Client {
    httpClient := resty.New()
    httpClient.SetDebug(true)
    //set timeout
    httpClient.SetTimeout(5 * time.Minute)

    //set retirees to 2
    httpClient.RetryCount = 2

    httpClient.SetAuthScheme("Bearer").SetAuthToken(bt)

    return httpClient
}

The http bin response indeed has content-length header. But the request does not have one

2020/12/08 08:42:42.174303 DEBUG RESTY 
==============================================================================
~~~ REQUEST ~~~
PATCH  /anything?action=flush&position=11111  HTTP/1.1
HOST   : httpbin.org
HEADERS:
    Authorization: Bearer test-token
    User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
PROTO        : HTTP/2.0
RECEIVED AT  : 2020-12-08T08:42:42.1673033+05:30
TIME DURATION: 1.2824316s
HEADERS      :
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Origin: *
    Content-Length: 544
    Content-Type: application/json
    Date: Tue, 08 Dec 2020 03:12:42 GMT
    Server: gunicorn/19.9.0
BODY         :
{
   "args": {
      "action": "flush",
      "position": "11111"
   },
   "data": "",
   "files": {},
   "form": {},
   "headers": {
      "Accept-Encoding": "gzip",
      "Authorization": "Bearer test-token",
      "Content-Length": "0",
      "Host": "httpbin.org",
      "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)",
      "X-Amzn-Trace-Id": "Root=1-5fceef2a-18ab2d1a6c6dc5e808d99102"
   },
   "json": null,
   "method": "PATCH",
   "origin": "x.x.x.x",
   "url": "https://httpbin.org/anything?action=flush&position=11111"
}

==============================================================================

as you can see the request does not have the content-length header

vijayrajah commented 3 years ago

one more interesting info.

if i set

SetContentLength(true)

// add header
reqHeader := map[string]string {
        "Content-Length": "0",
    }
request.SetHeaders(reqHeader)

resty debug shows the content-length header is being set

2020/12/08 09:02:06.212107 DEBUG RESTY 
==============================================================================
~~~ REQUEST ~~~
PATCH  /anything?action=flush&position=11111  HTTP/1.1
HOST   : httpbin.org
HEADERS:
    Authorization: Bearer test-token
    Content-Length: 0
    User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
    X-Ms-Version: 2019-12-12
BODY   :
***** NO CONTENT *****

But fidler trace does not show the header as part of this request....

PATCH https://httpbin.org/anything?action=flush&position=11111 HTTP/1.1
Host: httpbin.org
User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
Authorization: Bearer test-token
Accept-Encoding: gzip
moorereason commented 3 years ago

Resty debug likely won't include the Content-Length header unless you add the header yourself.

It looks to me like httpbin is receiving a Content-Length header (based upon your first reply). Can you cross-reference the httpbin output and Fiddler?

vijayrajah commented 3 years ago

@moorereason It seems this is inconsistent.. I do not see the Content-Length response from httpbin

here is the code I used 

package main

import (
    "fmt"
    "time"

    "github.com/go-resty/resty/v2"
)

func main() {
    client := resty.New().SetTimeout(5 * time.Minute).SetAuthScheme("Bearer").SetAuthToken("bt")
    client.RetryCount = 3

    resp, err := client.R().Patch("http://httpbin.org/anything")
    if err != nil {
        panic(err)
    }

    fmt.Printf("%s\n", resp.Body())
}

Here is the response from httpbin

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer bt", 
    "Host": "httpbin.org", 
    "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)", 
    "X-Amzn-Trace-Id": "Root=1-5fd44c6c-014416c57be5712a6459e40e"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}

While this was running I did packet capture... Here is the packet capture... ( Using: tcpdump -nn -vv -A port 80 )

10:21:56.489359 IP (tos 0x0, ttl 64, id 34255, offset 0, flags [DF], proto TCP (6), length 212)
    192.168.10.176.46818 > 52.3.32.149.80: Flags [P.], cksum 0x20b7 (incorrect -> 0x0fcf), seq 1:161, ack 1, win 502, options [nop,nop,TS val 768094857 ecr 114680685], length 160: HTTP
E.....@.@..d..
.4. ....PW....,(..... ......
-.2....mPATCH /anything HTTP/1.1
Host: httpbin.org
User-Agent: go-resty/2.3.0 (https://github.com/go-resty/resty)
Authorization: Bearer bt
Accept-Encoding: gzip

10:21:56.712734 IP (tos 0x0, ttl 220, id 51969, offset 0, flags [DF], proto TCP (6), length 52)
    52.3.32.149.80 > 192.168.10.176.46818: Flags [.], cksum 0x864b (correct), seq 1, ack 161, win 110, options [nop,nop,TS val 114680741 ecr 768094857], length 0
E..4..@.....4. ...
..P...,(.W......n.K.....
....-.2.
10:21:56.714495 IP (tos 0x0, ttl 220, id 51970, offset 0, flags [DF], proto TCP (6), length 282)
    52.3.32.149.80 > 192.168.10.176.46818: Flags [P.], cksum 0x0026 (correct), seq 1:231, ack 161, win 110, options [nop,nop,TS val 114680741 ecr 768094857], length 230: HTTP, length: 230
    HTTP/1.1 200 OK
    Date: Sat, 12 Dec 2020 04:51:56 GMT
    Content-Type: application/json
    Content-Length: 429
    Connection: keep-alive
    Server: gunicorn/19.9.0
    Access-Control-Allow-Origin: *
    Access-Control-Allow-Credentials: true

E.....@.....4. ...
..P...,(.W......n.&.....
....-.2.HTTP/1.1 200 OK
Date: Sat, 12 Dec 2020 04:51:56 GMT
Content-Type: application/json
Content-Length: 429
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

10:21:56.714523 IP (tos 0x0, ttl 64, id 34256, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.10.176.46818 > 52.3.32.149.80: Flags [.], cksum 0x2017 (incorrect -> 0x82fd), seq 161, ack 231, win 501, options [nop,nop,TS val 768095082 ecr 114680741], length 0
E..4..@.@.....
.4. ....PW....,)..... ......
-.3j....
10:21:56.714536 IP (tos 0x0, ttl 220, id 51971, offset 0, flags [DF], proto TCP (6), length 481)
    52.3.32.149.80 > 192.168.10.176.46818: Flags [P.], cksum 0xde76 (correct), seq 231:660, ack 161, win 110, options [nop,nop,TS val 114680741 ecr 768094857], length 429: HTTP
E.....@...."4. ...
..P...,).W......n.v.....
....-.2.{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer bt", 
    "Host": "httpbin.org", 
    "User-Agent": "go-resty/2.3.0 (https://github.com/go-resty/resty)", 
    "X-Amzn-Trace-Id": "Root=1-5fd44c6c-014416c57be5712a6459e40e"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}

If it makes any difference, This test I performed in my personal PC (go 1.15.6 on Linux ). The previous test was from my Work PC( go 1.15.4 & Win 10)

Is there a way to Force 'Content-length' and not let golang 'swallow' it?

vijayrajah commented 3 years ago

One more info.. Courtesy of this SO question

i wrote this code

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {

    url := "http://httpbin.org/anything"
    method := "PATCH"

    client := &http.Client{}
    req, err := http.NewRequest(method, url, http.NoBody)
    if err != nil {
        panic(err)
    }

    req.TransferEncoding = []string{"identity"}
    req.Header.Set("Authorization", "Bearer my-token")

    res, err := client.Do(req)
    defer res.Body.Close()
    body, err := ioutil.ReadAll(res.Body)

    fmt.Println(string(body))
}

For this I DO see the content-Length header

Response:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer my-token", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "Go-http-client/1.1", 
    "X-Amzn-Trace-Id": "Root=1-5fd463ac-40a22b297a343fc255e0e621"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}

Process finished with exit code 0

Packet capture

12:01:08.520958 IP (tos 0x0, ttl 64, id 21336, offset 0, flags [DF], proto TCP (6), length 205)
    192.168.10.176.37488 > 184.72.216.47.80: Flags [P.], cksum 0x5c90 (incorrect -> 0x123b), seq 1:154, ack 1, win 502, options [nop,nop,TS val 2495356391 ecr 304828402], length 153: HTTP
E...SX@.@.....
..H./.p.P.......*....\......
.....+O.PATCH /anything HTTP/1.1
Host: httpbin.org
User-Agent: Go-http-client/1.1
Content-Length: 0
Authorization: Bearer my-token
Accept-Encoding: gzip

12:01:08.724087 IP (tos 0x0, ttl 224, id 51983, offset 0, flags [DF], proto TCP (6), length 52)
    184.72.216.47.80 > 192.168.10.176.37488: Flags [.], cksum 0x5c08 (correct), seq 1, ack 154, win 110, options [nop,nop,TS val 304828453 ecr 2495356391], length 0
E..4..@...s..H./..
..P.p...*...q...n\......
.+P%....
12:01:08.725423 IP (tos 0x0, ttl 224, id 51984, offset 0, flags [DF], proto TCP (6), length 282)
    184.72.216.47.80 > 192.168.10.176.37488: Flags [P.], cksum 0xe2df (correct), seq 1:231, ack 154, win 110, options [nop,nop,TS val 304828453 ecr 2495356391], length 230: HTTP, length: 230
    HTTP/1.1 200 OK
    Date: Sat, 12 Dec 2020 06:31:08 GMT
    Content-Type: application/json
    Content-Length: 431
    Connection: keep-alive
    Server: gunicorn/19.9.0
    Access-Control-Allow-Origin: *
    Access-Control-Allow-Credentials: true

E.....@...r..H./..
..P.p...*...q...n.......
.+P%....HTTP/1.1 200 OK
Date: Sat, 12 Dec 2020 06:31:08 GMT
Content-Type: application/json
Content-Length: 431
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

12:01:08.725430 IP (tos 0x0, ttl 64, id 21337, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.10.176.37488 > 184.72.216.47.80: Flags [.], cksum 0x5bf7 (incorrect -> 0x58ce), seq 154, ack 231, win 501, options [nop,nop,TS val 2495356596 ecr 304828453], length 0
E..4SY@.@.....
..H./.p.P...q........[......
.....+P%
12:01:08.725434 IP (tos 0x0, ttl 224, id 51985, offset 0, flags [DF], proto TCP (6), length 483)
    184.72.216.47.80 > 192.168.10.176.37488: Flags [P.], cksum 0xb3a9 (correct), seq 231:662, ack 154, win 110, options [nop,nop,TS val 304828453 ecr 2495356391], length 431: HTTP
E.....@...r2.H./..
..P.p.......q...n.......
.+P%....{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Authorization": "Bearer my-token", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "Go-http-client/1.1", 
    "X-Amzn-Trace-Id": "Root=1-5fd463ac-40a22b297a343fc255e0e621"
  }, 
  "json": null, 
  "method": "PATCH", 
  "origin": "<MYIP>", 
  "url": "http://httpbin.org/anything"
}

Can we do similar thing in Resty?

moorereason commented 3 years ago

Interesting. If you hit the SSL URL (https://httpbin.org/anything), resty sends a Content-Length header. Non-SSL requests don't send the header. 😕 Hmm

moorereason commented 3 years ago

Findings

Ok. I'm not able to come up with a way to send a Content-Length header for non-http2 requests with the current release of resty (v2.3.0). My previous comment about SSL requests working was really that the client was using http2. If you disable http2 on the client (GODEBUG=http2client=0 go run main.go), the SSL request also fails to send the Content-Length header.

Work-around in master

With the most recent commit to master, the following will send a Content-Length header:

resp, err := resty.New().R().SetBody(http.NoBody).Patch("http://httpbin.org/anything")

However, that's not very intuitive and requires knowing a rather peculiar implementation detail with the http package.

Possible Fix

Given the following patch against master:

diff --git a/middleware.go b/middleware.go
index 50ff448..2b653fd 100644
--- a/middleware.go
+++ b/middleware.go
@@ -162,6 +162,12 @@ func createHTTPRequest(c *Client, r *Request) (err error) {
        if r.bodyBuf == nil {
                if reader, ok := r.Body.(io.Reader); ok {
                        r.RawRequest, err = http.NewRequest(r.Method, r.URL, reader)
+               } else if c.setContentLength || r.setContentLength {
+                       r.RawRequest, err = http.NewRequest(r.Method, r.URL, http.NoBody)
                } else {
                        r.RawRequest, err = http.NewRequest(r.Method, r.URL, nil)
                }

I'm able to get a Content-Length header on non-http2 requests when using only SetContentLength(true):

resp, err := resty.New().R().SetContentLength(true).Patch("http://httpbin.org/anything")

Because of what happens later on in createHTTPRequest, simply creating a NewRequest with nil doesn't maintain the body in a way that the http package can detect the zero-length body. I think this patch lets SetContentLength do it's job when we want it to without further complicating how we process the body.

@jeevatkm, what are your thoughts on this change? All existing tests pass.

moorereason commented 3 years ago

BTW, this is fixed upstream in Go 1.16beta1. See https://github.com/golang/go/issues/40978. The draft release notes say:

The Client now sends an explicit Content-Length: 0 header in PATCH requests with empty bodies, matching the existing behavior of POST and PUT.

With Go 1.16beta1 and resty v2.3.0, I'm now seeing a Content-Length: 0 header with the following request:

resty.New().R().Patch("http://httpbin.org/anything")

Yea! :tada:

vijayrajah commented 3 years ago

@moorereason Thanks a lot. That is the exact API, I'm trying to use. :) (azure DFS Gen2 API)

one more reason to look forward for 1.16 ( as it also includes //go:embed)

jeevatkm commented 3 years ago

@moorereason @vijayrajah Nice interaction 👍

@moorereason your fix suggestion, I think we can go for it.

jeevatkm commented 3 years ago

@moorereason I have applied your suggestion will be part of v2.4.0 release. @vijayrajah FYI

vijayrajah commented 3 years ago

@moorereason & @jeevatkm Thanks...

ramilmsh commented 1 year ago

@jeevatkm I cannot seem get this functionality to work

in go i make a call like this, combining every single way to set the content-length i know

    resty.New().
        SetBaseURL("http://127.0.0.1:8000").
        SetPreRequestHook(func(c *resty.Client, r *http.Request) error {
            r.TransferEncoding = []string{"identity"}
            r.Header.Set("Content-Length", "0")
            return nil
        }).
        SetContentLength(true).R().
        SetHeader("X-Test", "test").
        SetHeaders(map[string]string{"Content-Length": "0"}).
        SetContentLength(true).
        Post("/")

then using python's barebones server handler

import http.server

class Handler(http.server.SimpleHTTPRequestHandler):
    def do_POST(self) -> None:
        print(self.headers)
        print(self.headers["Content-Length"])
        self.send_response(200)

if __name__ == "__main__":
    server_address = ("", 8000)
    httpd = http.server.HTTPServer(server_address, Handler)
    httpd.serve_forever()

and i still get no content length

127.0.0.1 - - [06/Oct/2023 14:09:52] "POST /api/v2/account_balances/ HTTP/1.1" 200 -
Host: 127.0.0.1:8000
User-Agent: go-resty/2.9.1 (https://github.com/go-resty/resty)
X-Test: test
Accept-Encoding: gzip

None

I work around this by adding a minimal body, but it's weird... What am i missing?