go-rod / rod

A Chrome DevTools Protocol driver for web automation and scraping.
https://go-rod.github.io
MIT License
5.43k stars 357 forks source link

Launcher with custom http.Client #1121

Open rkuska opened 1 month ago

rkuska commented 1 month ago

Rod Version: v0.116.2

The code to demonstrate your question

I currently run headless-browser in google cloud run service using the go-rod image. From other service I connect to it via

launcher.NewManaged(browserURL)

What you got

The problem is that this request is executed by default http.Get request and it fails because it is not authorized (missing headers). So every call from my other service ends up with 403 when trying to setup the launcher and create a browser.

What you expect to see

I would expect to be able to inject a custom http.Client that would handle the authorization.

What have you tried to solve the question

I've tried navigating the code but it seems like the http.Get is hardcoded.

ysmood commented 1 month ago
package main

import (
    "context"

    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/cdp"
    "github.com/go-rod/rod/lib/launcher"
)

func main() {
    l := launcher.MustNewManaged("")

    u, h := l.ClientHeader()

    h.Set("My-Header", "my-header-value")

    client, err := cdp.StartWithURL(context.Background(), u, h)
    if err != nil {
        panic(err)
    }

    browser := rod.New().Client(client).MustConnect()

    browser.MustVersion()
}
github-actions[bot] commented 1 month ago

Please fix the format of your markdown:

7 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]

generated by check-issue

rkuska commented 1 month ago
package main

import (
  "context"

  "github.com/go-rod/rod"
  "github.com/go-rod/rod/lib/cdp"
  "github.com/go-rod/rod/lib/launcher"
)

func main() {
  l := launcher.MustNewManaged("")

  u, h := l.ClientHeader()

  h.Set("My-Header", "my-header-value")

  client, err := cdp.StartWithURL(context.Background(), u, h)
  if err != nil {
      panic(err)
  }

  browser := rod.New().Client(client).MustConnect()

  browser.MustVersion()
}

The problem is that following code:

    res, err := http.Get(toHTTP(*u).String()) //nolint: noctx
    if err != nil {
        return nil, err
    }

Is part of the call to MustNewManaged and this will fail even before setting the headers.

https://github.com/go-rod/rod/blob/main/lib/launcher/manager.go#L54

ysmood commented 1 month ago

I see, then we need a new method to support this requirement.

ysmood commented 1 month ago

Before the new version of rod, you might have to use a proxy to handle the extra http header for your service.