go-rod / stealth

anti-bot-detection with rod
https://pkg.go.dev/github.com/go-rod/stealth
MIT License
244 stars 31 forks source link

在遇到瑞数的情况,stealth无法起作用,将一直卡住 #28

Open cdsjsz opened 1 year ago

cdsjsz commented 1 year ago

复现代码如下

package main

import (
    "crypto/md5"
    "fmt"
    "github.com/go-rod/rod/lib/launcher"
    "time"

    "github.com/go-rod/rod"
    "github.com/go-rod/stealth"
)

func main() {
    path, _ := launcher.LookPath()
    url := launcher.New().Bin(path).Headless(false).MustLaunch()
    browser := rod.New().
        ControlURL(url).Timeout(time.Minute).MustConnect()
    defer browser.MustClose()

    // You can also use stealth.JS directly without rod
    fmt.Printf("js: %x\n\n", md5.Sum([]byte(stealth.JS)))

    page := stealth.MustPage(browser)

    page.MustNavigate("http://lib.cqvip.com/")
    time.Sleep(10 * time.Minute)

}
cdsjsz commented 1 year ago

newusermode也同样无法绕过检测

package main

import (
    "crypto/md5"
    "fmt"
    "github.com/go-rod/rod/lib/launcher"
    "time"

    "github.com/go-rod/rod"
    "github.com/go-rod/stealth"
)

func main() {
    path, _ := launcher.LookPath()
    url := launcher.NewUserMode().Bin(path).Headless(false).MustLaunch()
    browser := rod.New().
        ControlURL(url).Timeout(time.Minute).MustConnect()
    defer browser.MustClose()

    // You can also use stealth.JS directly without rod
    fmt.Printf("js: %x\n\n", md5.Sum([]byte(stealth.JS)))

    //page := stealth.MustPage(browser)
    page := browser.MustPage()

    page.MustNavigate("https://lib.cqvip.com/")
    time.Sleep(10 * time.Minute)

}
ysmood commented 1 year ago

你不需要 stealth

https://github.com/go-rod/rod/issues/744

package main

import (
    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/launcher"
    "github.com/go-rod/rod/lib/utils"
)

func main() {
    u := launcher.NewUserMode().
        Leakless(true).
        UserDataDir("tmp/t").
        Set("disable-default-apps").
        Set("no-first-run").
        MustLaunch()

    rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://lib.cqvip.com/")

    utils.Pause()
}
cdsjsz commented 1 year ago

感谢您的回复,在有头模式下没有问题,但在无头模式下还是会卡住,请问如果想要在无头模式下不卡白屏,应该如何操作呢,谢谢

ysmood commented 1 year ago

可以利用 docker 加 XVFB 来实现有头

https://github.com/go-rod/rod/blob/927baba50a3086dcc82906b585d98b182318a677/lib/examples/launch-managed/main.go#L28

cdsjsz commented 1 year ago

可以利用 docker 加 XVFB 来实现有头

https://github.com/go-rod/rod/blob/927baba50a3086dcc82906b585d98b182318a677/lib/examples/launch-managed/main.go#L28

感谢您的帮助,在docker模式下也会卡白屏,复现代码如下

package main

import (
    "fmt"

    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/launcher"
    "github.com/go-rod/rod/lib/utils"
)

func main() {

    l := launcher.MustNewManaged("")
    // Launch with headful mode
    l.Headless(false).XVFB("--server-num=5", "--server-args=-screen 0 1600x900x16").Set("disable-default-apps").
        Set("no-first-run").Set("disable-gpu")

    browser := rod.New().Client(l.MustClient()).MustConnect()

    // You may want to start a server to watch the screenshots of the remote browser.
    launcher.Open(browser.ServeMonitor(""))
    page := browser.MustPage("https://lib.cqvip.com/")
    fmt.Println(
        page.HTML(),
    )

    utils.Pause()
}

可能是由于不是newusermode模式下启动浏览器导致的?在docker模式下,我们能让他以newusermode启动么,谢谢

cdsjsz commented 1 year ago

如果仅将manager的改为newusermode,则返回websocket bad handshake: 502 Bad Gateway. 图片

ysmood commented 1 year ago
l.Flags = launcher.NewUserMode().Flags
cdsjsz commented 1 year ago
l.Flags = launcher.NewUserMode().Flags

感谢您的回复,加入该参数后会发生panic,显示:websocket bad handshake: 502 Bad Gateway. 复现代码如下

package main

import (
    "fmt"

    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/launcher"
    "github.com/go-rod/rod/lib/utils"
)

func main() {
    l := launcher.MustNewManaged("wss://example.com")
    l.Headless(false).XVFB("--server-num=5", "--server-args=-screen 0 1600x900x16").Set("disable-default-apps").
        Set("no-first-run").Set("disable-gpu")
    l.Flags = launcher.NewUserMode().Flags

    browser := rod.New().Client(l.MustClient()).MustConnect()

    // You may want to start a server to watch the screenshots of the remote browser.
    launcher.Open(browser.ServeMonitor(""))
    page := browser.MustPage("https://lib.cqvip.com/")
    fmt.Println(
        page.HTML(),
    )

    utils.Pause()
}
ysmood commented 1 year ago

你可以自己慢慢调试 flag,不过大概率是不行的。建议你提交 ticket 给 chromium 团队来处理。这已经超出 rod 应该处理的范围了。

cdsjsz commented 1 year ago

你可以自己慢慢调试 flag,不过大概率是不行的。建议你提交 ticket 给 chromium 团队来处理。这已经超出 rod 应该处理的范围了。

请问一下这是chromium哪方面出了问题啊?看这个报错怎么像是ssl的问题

ysmood commented 1 year ago

NewUserMode 部分的评论可以都忽略,这本来就不是 rod 的合理用法,只是可能能成功的 hack,不应该寄希望于此。 我意思是这个应该 work,但是白屏了,这是 chromium 的 bug:https://github.com/go-rod/stealth/issues/28#issuecomment-1558435303

你就算不用 rod,也会白屏的,这显然就不是 rod 的问题了。

cdsjsz commented 1 year ago

感谢您的解答,主要问题是该网站在正常浏览器下没有问题,NewUserMode有头模式下也没有问题,在New下无论有头无头都卡白屏,能否请教一下NewUserMode和New的关键差异在哪么?我尝试看了下相关的代码,主要是flag的不同?

ysmood commented 1 year ago

是的,我前面的回答也提到了

你可以自己慢慢调试 flag

cdsjsz commented 1 year ago

好的,感谢您的帮助