golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
122.29k stars 17.47k forks source link

net/http: cookies not set on redirections (302) by the default client #64464

Open Amodio opened 9 months ago

Amodio commented 9 months ago

Go version

go version go1.19.8 linux/amd64

What operating system and processor architecture are you using (go env)?

GOVERSION="go1.19.8"
GOHOSTARCH="amd64"
GOHOSTOS="linux"

What did you do?

First case, I did a GET onto a webpage that sets a cookie and redirects users: https://go.dev/play/p/ecskkzjvHRy Second case, I did a POST onto a webpage that sets a cookie and redirects users: https://go.dev/play/p/a4xWVv0ZVqT

What did you expect to see?

In both cases, I expected a redirection to a webpage with cookies set. It turns out that a Cookie Jar has to be defined/initialized for this behaviour to happen (see here for example). Corrected code for each case: GET & POST. Output in the first case:

$ go run main.go
Response: Cookie was: SET<br><a href="?a=1701347614.2247">try again</a>
Cookie: test=deleted; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Max-Age=0

In the second case, I got a Wrong Credentials error.

What did you see instead?

First case:

$ go run main.go
Response: Cookie was: NOT SET<br><a href="?a=1701347518.7983">try again</a>
Cookie: test=deleted; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Max-Age=0

Second case detailed log (censored):

$ GODEBUG=http2debug=2 go run main.go
2023/11/30 12:28:37 http2: Transport failed to get client conn for tictoc2.chall.malicecyber.com:443: http2: no cached connection was available
2023/11/30 12:28:37 http2: Transport creating client conn 0xc00010c180 to 46.30.202.223:443
2023/11/30 12:28:37 http2: Framer 0xc000068e00: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2023/11/30 12:28:37 http2: Framer 0xc000068e00: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2023/11/30 12:28:37 http2: Transport encoding header ":authority" = "tictoc2.chall.malicecyber.com"
2023/11/30 12:28:37 http2: Transport encoding header ":method" = "POST"
2023/11/30 12:28:37 http2: Transport encoding header ":path" = "/login.php"
2023/11/30 12:28:37 http2: Transport encoding header ":scheme" = "https"
2023/11/30 12:28:37 http2: Transport encoding header "content-type" = "application/x-www-form-urlencoded"
2023/11/30 12:28:37 http2: Transport encoding header "content-length" = "81"
2023/11/30 12:28:37 http2: Transport encoding header "accept-encoding" = "gzip"
2023/11/30 12:28:37 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2023/11/30 12:28:37 http2: Framer 0xc000068e00: wrote HEADERS flags=END_HEADERS stream=1 len=84
2023/11/30 12:28:37 http2: Framer 0xc000068e00: wrote DATA flags=END_STREAM stream=1 len=81 data="password=X&username=admin"
2023/11/30 12:28:37 http2: Framer 0xc000068e00: read SETTINGS len=24, settings: MAX_FRAME_SIZE=1048576, MAX_CONCURRENT_STREAMS=250, MAX_HEADER_LIST_SIZE=1048896, INITIAL_WINDOW_SIZE=1048576
2023/11/30 12:28:37 http2: Transport received SETTINGS len=24, settings: MAX_FRAME_SIZE=1048576, MAX_CONCURRENT_STREAMS=250, MAX_HEADER_LIST_SIZE=1048896, INITIAL_WINDOW_SIZE=1048576
2023/11/30 12:28:37 http2: Framer 0xc000068e00: wrote SETTINGS flags=ACK len=0
2023/11/30 12:28:37 http2: Framer 0xc000068e00: read WINDOW_UPDATE len=4 (conn) incr=983041
2023/11/30 12:28:37 http2: Transport received WINDOW_UPDATE len=4 (conn) incr=983041
2023/11/30 12:28:37 http2: Framer 0xc000068e00: read SETTINGS flags=ACK len=0
2023/11/30 12:28:37 http2: Transport received SETTINGS flags=ACK len=0
2023/11/30 12:28:37 http2: Framer 0xc000068e00: read WINDOW_UPDATE len=4 (conn) incr=81
2023/11/30 12:28:37 http2: Transport received WINDOW_UPDATE len=4 (conn) incr=81
2023/11/30 12:28:48 http2: Framer 0xc000068e00: read HEADERS flags=END_STREAM|END_HEADERS stream=1 len=205
2023/11/30 12:28:48 http2: decoded hpack field header field ":status" = "302"
2023/11/30 12:28:48 http2: decoded hpack field header field "cache-control" = "no-store, no-cache, must-revalidate"
2023/11/30 12:28:48 http2: decoded hpack field header field "content-type" = "text/html; charset=UTF-8"
2023/11/30 12:28:48 http2: decoded hpack field header field "date" = "Thu, 30 Nov 2023 11:28:37 GMT"
2023/11/30 12:28:48 http2: decoded hpack field header field "expires" = "Thu, 19 Nov 1981 08:52:00 GMT"
2023/11/30 12:28:48 http2: decoded hpack field header field "location" = "/account.php"
2023/11/30 12:28:48 http2: decoded hpack field header field "pragma" = "no-cache"
2023/11/30 12:28:48 http2: decoded hpack field header field "server" = "Apache/2.4.56 (Debian)"
2023/11/30 12:28:48 http2: decoded hpack field header field "set-cookie" = "PHPSESSID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; path=/"
2023/11/30 12:28:48 http2: decoded hpack field header field "x-powered-by" = "PHP/8.2.6"
2023/11/30 12:28:48 http2: decoded hpack field header field "content-length" = "0"
2023/11/30 12:28:48 http2: Transport received HEADERS flags=END_STREAM|END_HEADERS stream=1 len=205
2023/11/30 12:28:48 http2: Transport encoding header ":authority" = "tictoc2.chall.malicecyber.com"
2023/11/30 12:28:48 http2: Transport encoding header ":method" = "GET"
2023/11/30 12:28:48 http2: Transport encoding header ":path" = "/account.php"
2023/11/30 12:28:48 http2: Transport encoding header ":scheme" = "https"
2023/11/30 12:28:48 http2: Transport encoding header "referer" = "https://tictoc2.chall.malicecyber.com/login.php"
2023/11/30 12:28:48 http2: Transport encoding header "content-type" = "application/x-www-form-urlencoded"
2023/11/30 12:28:48 http2: Transport encoding header "accept-encoding" = "gzip"
2023/11/30 12:28:48 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2023/11/30 12:28:48 http2: Framer 0xc000068e00: wrote HEADERS flags=END_STREAM|END_HEADERS stream=3 len=52
2023/11/30 12:28:49 http2: Framer 0xc000068e00: read HEADERS flags=END_STREAM|END_HEADERS stream=3 len=98
2023/11/30 12:28:49 http2: decoded hpack field header field ":status" = "302"
2023/11/30 12:28:49 http2: decoded hpack field header field "cache-control" = "no-store, no-cache, must-revalidate"
2023/11/30 12:28:49 http2: decoded hpack field header field "content-type" = "text/html; charset=UTF-8"
2023/11/30 12:28:49 http2: decoded hpack field header field "date" = "Thu, 30 Nov 2023 11:28:48 GMT"
2023/11/30 12:28:49 http2: decoded hpack field header field "expires" = "Thu, 19 Nov 1981 08:52:00 GMT"
2023/11/30 12:28:49 http2: decoded hpack field header field "location" = "index.php?error=wrong_credentials"
2023/11/30 12:28:49 http2: decoded hpack field header field "pragma" = "no-cache"
2023/11/30 12:28:49 http2: decoded hpack field header field "server" = "Apache/2.4.56 (Debian)"
2023/11/30 12:28:49 http2: decoded hpack field header field "set-cookie" = "PHPSESSID=68879433fa243e2e9cfae6f7a8fed841; path=/"
2023/11/30 12:28:49 http2: decoded hpack field header field "x-powered-by" = "PHP/8.2.6"
2023/11/30 12:28:49 http2: decoded hpack field header field "content-length" = "0"
2023/11/30 12:28:49 http2: Transport received HEADERS flags=END_STREAM|END_HEADERS stream=3 len=98
2023/11/30 12:28:49 http2: Transport encoding header ":authority" = "tictoc2.chall.malicecyber.com"
2023/11/30 12:28:49 http2: Transport encoding header ":method" = "GET"
2023/11/30 12:28:49 http2: Transport encoding header ":path" = "/index.php?error=wrong_credentials"
2023/11/30 12:28:49 http2: Transport encoding header ":scheme" = "https"
2023/11/30 12:28:49 http2: Transport encoding header "content-type" = "application/x-www-form-urlencoded"
2023/11/30 12:28:49 http2: Transport encoding header "referer" = "https://tictoc2.chall.malicecyber.com/account.php"
2023/11/30 12:28:49 http2: Transport encoding header "accept-encoding" = "gzip"
2023/11/30 12:28:49 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2023/11/30 12:28:49 http2: Framer 0xc000068e00: wrote HEADERS flags=END_STREAM|END_HEADERS stream=5 len=69
2023/11/30 12:28:49 http2: Framer 0xc000068e00: read HEADERS flags=END_HEADERS stream=5 len=28
2023/11/30 12:28:49 http2: decoded hpack field header field ":status" = "200"
2023/11/30 12:28:49 http2: decoded hpack field header field "content-encoding" = "gzip"
2023/11/30 12:28:49 http2: decoded hpack field header field "content-type" = "text/html; charset=UTF-8"
2023/11/30 12:28:49 http2: decoded hpack field header field "date" = "Thu, 30 Nov 2023 11:28:48 GMT"
2023/11/30 12:28:49 http2: decoded hpack field header field "server" = "Apache/2.4.56 (Debian)"
2023/11/30 12:28:49 http2: decoded hpack field header field "vary" = "Accept-Encoding"
2023/11/30 12:28:49 http2: decoded hpack field header field "x-powered-by" = "PHP/8.2.6"
2023/11/30 12:28:49 http2: decoded hpack field header field "content-length" = "972"
2023/11/30 12:28:49 http2: Transport received HEADERS flags=END_HEADERS stream=5 len=28
2023/11/30 12:28:49 http2: Framer 0xc000068e00: read DATA flags=END_STREAM stream=5 len=972 data="\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x9dV\xdbn\xdb8\x10}\xcfWp\x05\xecS!Ɏ\xedn\x9b\xda\xc6f\xdbf\x91\xc0\xf1&\xc8}\xdf(\x8a\x92\x98R$A\x8e\xec(_\xbfCIN\x9d\xf8\x82t\rX\xe6\x90\xc33s\xe6f\x1d\x102.\xa0\x94SB\xc8\x01~\xc9\xd8A-\xf9\xf4\xa0\x15\xe2Nj\x84\xdf\u0090\x9c2\xadHfuI\n\x00\xe3\x8e\xe2x\xb9\\F\x99\xa4 \xf0$b\xba\x8c3\xcby\xe8%\x17疪4\xa3Pp\x1b2\xa9ُ\xa0\xb3\xe2?I\xfe\n\xc3\x15\x15\x00\xb7\x0eP\xad\xc1\x11%\xcdyh\n\r:^\b\x05^ZR)[$\x17.\xa8\x15\x1c\xea\xb0qх\xc3a\xefp8\x18\xf5\x0f\x1b\x03a\xd89]p\x9ar;}\xb1:\x96B\xfd \x85\xe5\xd9$X\x99g\xa9\x8a\x1e]ʥX\xd8Hq\x88\x95)\xe3Dkp`\xa9\xf9s\x14\xf5\xa2\xc38\x15\x0eb\xe6\xdcσ\xa8\x14\xc8ع\x80X.'A" (716 bytes omitted)
2023/11/30 12:28:49 http2: Transport received DATA flags=END_STREAM stream=5 len=972 data="\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x9dV\xdbn\xdb8\x10}\xcfWp\x05\xecS!Ɏ\xedn\x9b\xda\xc6f\xdbf\x91\xc0\xf1&\xc8}\xdf(\x8a\x92\x98R$A\x8e\xec(_\xbfCIN\x9d\xf8\x82t\rX\xe6\x90\xc33s\xe6f\x1d\x102.\xa0\x94SB\xc8\x01~\xc9\xd8A-\xf9\xf4\xa0\x15\xe2Nj\x84\xdf\u0090\x9c2\xadHfuI\n\x00\xe3\x8e\xe2x\xb9\\F\x99\xa4 \xf0$b\xba\x8c3\xcby\xe8%\x17疪4\xa3Pp\x1b2\xa9ُ\xa0\xb3\xe2?I\xfe\n\xc3\x15\x15\x00\xb7\x0eP\xad\xc1\x11%\xcdyh\n\r:^\b\x05^ZR)[$\x17.\xa8\x15\x1c\xea\xb0qх\xc3a\xefp8\x18\xf5\x0f\x1b\x03a\xd89]p\x9ar;}\xb1:\x96B\xfd \x85\xe5\xd9$X\x99g\xa9\x8a\x1e]ʥX\xd8Hq\x88\x95)\xe3Dkp`\xa9\xf9s\x14\xf5\xa2\xc38\x15\x0eb\xe6\xdcσ\xa8\x14\xc8ع\x80X.'A" (716 bytes omitted)
Response: 
  <html>   
    <style>

    </style>
    <!-- Icon from https://www.flaticon.com/free-icons/grandfather-clock"  
        bg https://www.shutterstock.com/image-photo/vintage-wall-clocks-variety-styles-440243512
    -->
    <header>
        <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous">
        <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous"></script>
        <link href="./Styles/style.css" rel="stylesheet" ></link>
    </header>
    <body >
        <nav class="navbar navbar-light bg-light bannerTop">
             <span class="navbar-brand mb-0 h1"><img src="./images/clock.png" style="width:1em;"> TicToc</span>
             <span class="navbar-brand mb-0 h1"> <a href="contact.php"> <button type="button" class="btn btn-light"> Join Us </btn> </a> </span>
        </nav>

        <div style="height:auto">
            <div class="d-flex justify-content-center align-self-center">
                <fieldset>
                <div>
                    <div>
                        <h3> <i> No more time waster... </i></h3> 
                        <hr>
                        <h5>Use your connection Information to access your clocks</h4>
                        <br>
                    </div>

            <p ><h5 class="text-danger">Wrong credentials</h5></p>                <form method="post" action="login.php">
                    <div><h4>Login</h4></div>                 
                    <div><input type="text" name="username" /></div>
                    <div><h4> Password</h4></div>
                    <div><input type="password" name="password"/></div>
                    <div><input type="submit" class="btn btn-primary btn-sm mt-2" value="Login"></div>
                </form>
                </div>
                </fieldset>
            </div>
        </div>

    </body>

        <footer class="bannerBot">
                <div class="stage animate" style="height:100%">
                <div class="stage" style="height:100%;left:-100%">
                </div>
            </div>
        </footer>
    <html>

As pointed out in this 2012 blog post, most browsers redirect users by setting cookies and Golang does only half of this job by default. Please make this default, or do not redirect by default.

Amodio commented 9 months ago

It works as expected in Python:

$ cat test.py
import requests

response = requests.get("http://dubbelboer.com/302cookie.php")
if response.status_code == 200:
    print(response.text)
else:
    print(f"Error: {response.status_code}")
final_url = response.url
print(f"Final URL: {final_url}")

$ python3 test.py 
Cookie was: SET<br><a href="?a=1701349447.3061">try again</a>
Final URL: https://dubbelboer.com/302cookie.php?show=1701349447.2922
bcmills commented 9 months ago

This behavior is as documented. Per https://pkg.go.dev/net/http#Client:

When following redirects, the Client will forward all headers set on the initial Request except: … when forwarding the "Cookie" header with a non-nil cookie Jar. …

Amodio commented 9 months ago

Even if it is documented, Python and browsers handle it differently so, can that evolve please?

Furthermore, the documentation says the Golang HTTP (default) client forwards all headers set on the initial request (with exceptions), but here the cookie is set on the initial response, isn't it? I understand this: If Jar is nil, the initial cookies are forwarded without change. as: by default, cookies are forwarded (= it should work), is it wrong?

seankhliao commented 9 months ago

There'd be an import cycle if net/http tried to import net/http/cookiejar. It would also make the default client much easier to track.

cc @neild