go-rod / rod

A Chrome DevTools Protocol driver for web automation and scraping.
https://go-rod.github.io
MIT License
5.25k stars 345 forks source link

Missing Request Body When Loading "https://www.booking.com" Using go-rod #1018

Closed sensensen404 closed 6 months ago

sensensen404 commented 7 months ago

Rod Version: v0.114.7

I've encountered an issue when using go-rod to load "https://www.booking.com/" and hijack requests. Specifically, the problem arises with requests to the GraphQL API ("https://www.booking.com/dml/graphql"). While the browser's developer console shows that these requests include a request body, the intercepted request in go-rod appears to have a missing body, leading to request failure.

I'm not entirely sure if this issue is directly related to go-rod or if it's an underlying issue with the way requests are intercepted or processed. Any insights or suggestions on how to address this issue would be greatly appreciated.

The code to demonstrate your question

  1. Clone Rod to your local and cd to the repository:

    git clone https://github.com/go-rod/rod
    cd rod
  2. Replace the content of function Example_hijack_requests in file examples_test.go.

  3. Replace ALL THE CONTENT under Example_hijack_requests function, like below:

func Example_hijack_requests() {

    browser := rod.New().MustConnect()
    defer browser.MustClose()

    router := browser.HijackRequests()
    defer router.MustStop()

    router.MustAdd("*/graphql*", func(ctx *rod.Hijack) {
        url := ctx.Request.Req().URL.String()
        fmt.Println("ctx test:" + url)
        fmt.Println("ctx test body:" + ctx.Request.Body())

        _ = ctx.LoadResponse(http.DefaultClient, true)

        fmt.Println("ctx test response status:" + ctx.Response.RawResponse.Status)
    })

    go router.Run()

    browser.MustPage("https://www.booking.com").MustWaitDOMStable()

    fmt.Println("done")
    // Output: done
}

What you got

The output logs show the request URL but indicate an empty body for requests that definitely contain a body when inspected directly via the browser's developer console.

What you expect to see

The request body for GraphQL requests to "https://www.booking.com/dml/graphql" should have a body.

What have you tried to solve the question

github-actions[bot] commented 7 months ago

Please fix the format of your markdown:

61:1 MD033/no-inline-html Inline HTML [Element: img]
62:1 MD033/no-inline-html Inline HTML [Element: img]

generated by check-issue

ysmood commented 6 months ago

https://github.com/go-rod/rod/blob/0243a30b10871506324281942f9b16eeb45fdc12/hijack_test.go#L87