Can't access iframe on some websites

[ysmood]

Rod Version: v0.102.0

The code to demonstrate your question

package main

import (
    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/defaults"
)

func main() {
    defaults.Show = true

    page := rod.New().MustConnect().NoDefaultDevice().MustPage("https://captcha.website")
    f := page.MustElement("div:not([style*='display:']) > iframe[data-hcaptcha-widget-id]").MustFrame()
    f.MustElement("#checkbox")
}

The code above will panic. It works fine on headless mode.

[ysmood]

A temp solution is to use PageFromTargetID to create a standalone page for the iframe:

package main

import (
    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/defaults"
    "github.com/go-rod/rod/lib/proto"
    "github.com/go-rod/rod/lib/utils"
)

func main() {
    defaults.Show = true

    page := rod.New().MustConnect().NoDefaultDevice().MustPage("https://captcha.website")
    f := page.MustElement("div:not([style*='display:']) > iframe[data-hcaptcha-widget-id]").MustFrame()
    p := page.Browser().MustPageFromTargetID(proto.TargetTargetID(f.FrameID))
    p.MustElement("#checkbox").MustClick()
    utils.Pause()
}

[ysmood]

I think it's a bug of devtools, I don't know why Pierce: true can't get the content of the iframe:

https://github.com/go-rod/rod/blob/14ebb72947cc99e916ed97897473be2b6c64f39d/page_eval.go#L349

[KMACEL]

I encountered the same error. When I remove "show" from .rod, it works, but if the browser opens, it gets an error.

[ysmood]

I can confirm that puppeteer has the same issue, the code below will crash:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: false
  });
  const page = await browser.newPage();
  await page.goto('https://captcha.website')
  await new Promise((r) => setTimeout(r, 5000))
  const el = await page.$("div:not([style*='display:']) > iframe[data-hcaptcha-widget-id]")
  const frame = await el.contentFrame()
  await frame.$('#checkbox').click()
})();

[tlopo]

Since this is probably a bug on CDP, does anyone know if it's being tracked somewhere ?

[ysmood]

@tlopo I have opened an issue on puppeteer, but no update yet

[jhemmmm]

Any update on this?

Your temp solution didn't work on it me.

[timestee]

The issue https://github.com/puppeteer/puppeteer/issues/8150 has been confirmed but still no update yet~

[h3isenbug]

Stuck with the same issue here. FrameID workaround did not help.

[Samler-Lee]

A temp solution is to download an older version of Chromium and replace the current version of Chromium. I successfuly tested my code with on version 884014

[alicazi]

I'm getting "Command can only be executed on top-level targets" error when PageFromTarget(proto.TargetTargetID(f.FrameID)) which f is what I got from calling el.Frame(). Wondering is this a same issue? then why the error is so different? BTW, no success with other suggested solutions (not tried old Chromium version though as I don't see it as a sustainable solution). FYI, I'm trying to create a test that needs filling credit card information in Stripe iframe form.

[ysmood]

I will take some time to redesign how the js runtime works to completely resolve this issue.

[alicazi]

@ysmood Thanks

[hsxsix]

I'm getting "Command can only be executed on top-level targets" error when PageFromTarget(proto.TargetTargetID(f.FrameID)) which f is what I got from calling el.Frame(). Wondering is this a same issue? then why the error is so different? BTW, no success with other suggested solutions (not tried old Chromium version though as I don't see it as a sustainable solution). FYI, I'm trying to create a test that needs filling credit card information in Stripe iframe form.

same problem, any updates?

[zplzpl]

当 PageFromTarget(proto.TargetTargetID(f.FrameID)) 时出现“命令只能在顶级目标上执行”错误，其中 f 是我通过调用 el.Frame() 得到的。 想知道这是同一个问题吗？那么为什么错误如此不同呢？ 顺便说一句，其他建议的解决方案没有成功（没有尝试过旧的 Chromium 版本，因为我不认为它是一个可持续的解决方案）。 仅供参考，我正在尝试创建一个需要在 Stripe iframe 表单中填写信用卡信息的测试。

same problem

[zplzpl]

A temp solution is to download an older version of Chromium and replace the current version of Chromium. I successfuly tested my code with on version 884014

worked

[shynome]

A temp solution is to download an older version of Chromium and replace the current version of Chromium. I successfuly tested my code with on version 884014

the version 884014 is not working on debian 12.

Error code: 159

---

update: the version 884014 is working on debian 11. the problem may be libc version conflict

update: the version 884014 is not work with the latest go-rod

update: Puppeteer work as expect

[ctinkong]

v0.114.5 have same problem.

[rdelcampog]

Any progress? :(

[rgunindi]

didnt work any temp solution for me.

[cplasfwst]

I have the same problem. Dear author, do you have any solution?

[ysmood]

I'm refactoring the core code of this part, I will definitely progress it within this year.

[cplasfwst]

Thank you for your hard work. I also hope to solve it as soon as possible. Is there any temporary solution that can be used now?

[ysmood]

@cplasfwst you can follow this example to use the low-level cdp api to track the tree of targets (a target is like a iframe):

https://github.com/ysmood/rod/blob/0340433eab4e6551f075eaf1875035469a777480/examples_test.go#L410-L432

You have to use the apis list in this file:

https://github.com/ysmood/rod/blob/0340433eab4e6551f075eaf1875035469a777480/lib/proto/target.go

Then you can convert a target to a rod.Page with Browser.MustPageFromTargetID.

[cplasfwst]

Thank you very much for your prompt answer. I don't quite understand the case. If my structure is like this: xframe := page.Timeout(5 * time.Second).MustElementX(//iframe[@style="border: none; display: block; visibility: visible; border-radius: 6px; overflow: hidden; position: absolute; left: 50%; top: 50%; transform: translate(-50%, -50%); width: 380px; height: 384px;"]).MustFrame() fmt.Println("检测到有验证-----------------------------------------!!!!!!!!!!!!!!!!!!qwe", xframe) xframe.MustElementX(//canvas[@id="captcha_verify_image"]).MustScreenshot(uri +/ceshi.png)

May I ask if you could provide me with a case study using my code to make it easier for me to understand? Thank you very much. I hope you can help me

[cplasfwst]

Because the code above only works normally in headless mode, memory errors will appear when displayed

[ysmood]

Sorry, I can't provide more info, it's very hard to explain. If you want to solve it you have to work on your own.

[rgunindi]

@ysmood How can we use cdp? Any kind of documentation?

[ysmood]

@rgunindi https://github.com/go-rod/rod/issues/548#issuecomment-1983067842

[hunjixin]

headless also fail， but seem not all.

[alplf123]

c, err := launcher.New().
            NoSandbox(true).
            Set("disable-web-security").
            Set("disable-site-isolation-trials").
            Headless(false).
            Leakless(false).
            RemoteDebuggingPort(debugPort).
            Launch()

add disable-web-security disable-site-isolation-trials it works for me

[ysmood]

@alplf123 Thank you for the tips, it works for me for a simple test like below! I will try to create a unit test to reproduce it, if it works as expected, I will make it the default launch option.

The code like below will crash due to not able to access the iframe:

package main

import (
    "fmt"

    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/launcher"
)

func main() {
    u := launcher.New().
        Headless(false).
        MustLaunch()

    page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
    f := page.MustElement(`iframe[src*="https://challenges.cloudflare.com"]`).MustFrame()
    fmt.Println(f.MustElement("#success").MustHTML())
}

But the code like this will work:

package main

import (
    "fmt"

    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/launcher"
)

func main() {
    u := launcher.New().
        NoSandbox(true).
        Set("disable-web-security").
        Set("disable-site-isolation-trials").
        MustLaunch()

    page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
    f := page.MustElement(`iframe[src*="https://challenges.cloudflare.com"]`).MustFrame()
    fmt.Println(f.MustElement("#success").MustHTML())
}

The three flags are both critical:

        NoSandbox(true).
        Set("disable-web-security").
        Set("disable-site-isolation-trials").

[ysmood]

v0.116.1 has been released, please give it a try.

[Longdexin]

I've tried v0.116.1, it works as expected, but the cloudflare check progress was looping forever. When I returned to the older version v116.0, the cloudflare checkbox was checked and the page was redirected successfully.

[ysmood]

@Longdexin It works fine to me, I think it's because of your ip and browser fingerprint.

https://github.com/go-rod/rod/assets/1415488/0a78e1fa-a3f9-4759-b797-88d9e4ae8d73

[Longdexin]

@ysmood , yes, you got a point here. Thank you so much for your great work, it helps me a lot.

[Longdexin]

@ysmood hi, it's me again. I found that you had put flag "disable-site-isolation-trials" into defaultFlags, maybe this is the cause of the cloudflare check loop I mentioned earlier. When I wrote '.Delete("disable-site-isolation-trials")' to delete this flag, the problem was gone.

[ysmood]

@Longdexin #1076

[Longdexin]

@ysmood Hi, my code could no longer access the cloudflare iframe element, and after some digging, I found that the iframe had been using '#shadow-root (closed)' as its parent node recently, as described here https://developer.mozilla.org/en-US/docs/Web/API/ShadowRoot/mode. I managed to fix the test code somehow as below, hoping to help other users with the same problem.

package main

import (
    "fmt"
    "time"

    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/launcher"
)

func main() {
    u := launcher.New().
        Headless(false).
        NoSandbox(true).
        Set("disable-web-security").
        Set("disable-site-isolation-trials").
        MustLaunch()

    page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
    el, err := page.Timeout(30 * time.Second).Element(`div.cf-turnstile-wrapper`)
    if err != nil {
        panic(err)
    }
    el, err = el.ShadowRoot()
    if err != nil {
        panic(err)
    }
    _, err = el.Eval("()=>this.mode='open'")
    if err != nil {
        panic(err)
    }
    f := el.MustElement(`iframe[src*="https://challenges.cloudflare.com"]`).MustFrame()
    fmt.Println(f.MustElement("#success").MustHTML())
}

[moxcomic]

@ysmood Hi, my code could no longer access the cloudflare iframe element, and after some digging, I found that the iframe had been using '#shadow-root (closed)' as its parent node recently, as described here https://developer.mozilla.org/en-US/docs/Web/API/ShadowRoot/mode. I managed to fix the test code somehow as below, hoping to help other users with the same problem.

package main

import (
  "fmt"
  "time"

  "github.com/go-rod/rod"
  "github.com/go-rod/rod/lib/launcher"
)

func main() {
  u := launcher.New().
      Headless(false).
      NoSandbox(true).
      Set("disable-web-security").
      Set("disable-site-isolation-trials").
      MustLaunch()

  page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
  el, err := page.Timeout(30 * time.Second).Element(`div.cf-turnstile-wrapper`)
  if err != nil {
      panic(err)
  }
  el, err = el.ShadowRoot()
  if err != nil {
      panic(err)
  }
  _, err = el.Eval("()=>this.mode='open'")
  if err != nil {
      panic(err)
  }
  f := el.MustElement(`iframe[src*="https://challenges.cloudflare.com"]`).MustFrame()
  fmt.Println(f.MustElement("#success").MustHTML())
}

I tried using your code, but it failed to pass verification. When I manually click the verification checkbox, it spins for a few seconds and then returns to the unchecked state. This happens repeatedly, even after several attempts.

[Longdexin]

@moxcomic Hi, if you look at the html structure of this page (https://dash.cloudflare.com/sign-up) now, you can see it has two #shadow-root in the iframe, while it had only one when I posted the code above. I fixed the example code like below.

package main

import (
    "fmt"
    "time"

    "github.com/go-rod/rod"
    "github.com/go-rod/rod/lib/launcher"
    "github.com/go-rod/rod/lib/utils"
)

func main() {
    u := launcher.New().
        Headless(false).
        NoSandbox(true).
        Set("disable-web-security").
        Set("disable-site-isolation-trials").
        MustLaunch()

    page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
    el, err := page.Timeout(30 * time.Second).Element(`div.cf-turnstile-wrapper`)
    if err != nil {
        panic(err)
    }
    el, err = el.ShadowRoot()
    if err != nil {
        panic(err)
    }
    _, err = el.Eval("()=>this.mode='open'")
    if err != nil {
        panic(err)
    }
    f := el.MustElement(`iframe[src*="https://challenges.cloudflare.com"]`).MustFrame()
    utils.Sleep(10)
    el, err = f.MustElement("body").ShadowRoot()
    if err != nil {
        panic(err)
    }
    _, err = el.Eval("()=>this.mode='open'")
    if err != nil {
        panic(err)
    }
    fmt.Println(el.MustElement("#success").MustHTML())
}

However, 5 minutes later after I commented last time, I found the same repeating problem as you said and could not find any way to go. Sorry, the cloudflare thing is so disgusting, good luck to both of us.

[moxcomic]

@moxcomic Hi, if you look at the html structure of this page (https://dash.cloudflare.com/sign-up) now, you can see it has two #shadow-root in the iframe, while it had only one when I posted the code above. I fixed the example code like below.

package main

import (
  "fmt"
  "time"

  "github.com/go-rod/rod"
  "github.com/go-rod/rod/lib/launcher"
  "github.com/go-rod/rod/lib/utils"
)

func main() {
  u := launcher.New().
      Headless(false).
      NoSandbox(true).
      Set("disable-web-security").
      Set("disable-site-isolation-trials").
      MustLaunch()

  page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
  el, err := page.Timeout(30 * time.Second).Element(`div.cf-turnstile-wrapper`)
  if err != nil {
      panic(err)
  }
  el, err = el.ShadowRoot()
  if err != nil {
      panic(err)
  }
  _, err = el.Eval("()=>this.mode='open'")
  if err != nil {
      panic(err)
  }
  f := el.MustElement(`iframe[src*="https://challenges.cloudflare.com"]`).MustFrame()
  utils.Sleep(10)
  el, err = f.MustElement("body").ShadowRoot()
  if err != nil {
      panic(err)
  }
  _, err = el.Eval("()=>this.mode='open'")
  if err != nil {
      panic(err)
  }
  fmt.Println(el.MustElement("#success").MustHTML())
}

However, 5 minutes later after I commented last time, I found the same repeating problem as you said and could not find any way to go. Sorry, the cloudflare thing is so disgusting, good luck to both of us.

I found the problem. If Set("disable-site-isolation-trials") is added, it triggers this issue. Without it, verification passes normally, but it gives a runtime error: invalid memory address or nil pointer dereference error at el, err = f.MustElement("body").ShadowRoot().

[Longdexin]

A temp solution is to use PageFromTargetID to create a standalone page for the iframe:

package main

import (
  "github.com/go-rod/rod"
  "github.com/go-rod/rod/lib/defaults"
  "github.com/go-rod/rod/lib/proto"
  "github.com/go-rod/rod/lib/utils"
)

func main() {
  defaults.Show = true

  page := rod.New().MustConnect().NoDefaultDevice().MustPage("https://captcha.website")
  f := page.MustElement("div:not([style*='display:']) > iframe[data-hcaptcha-widget-id]").MustFrame()
  p := page.Browser().MustPageFromTargetID(proto.TargetTargetID(f.FrameID))
  p.MustElement("#checkbox").MustClick()
  utils.Pause()
}

@moxcomic maybe you should take this temp solution into consideration to avoid the runtime error. I'll be back here to share my code if I figure a way out in several days.

[moxcomic]

@moxcomic maybe you should take this temp solution into consideration to avoid the runtime error. I'll be back here to share my code if I figure a way out in several days.

@Longdexin Thank you. I was using it normally before, but recently, after some changes in CF, I can no longer access the checkbox, and I haven't found a good solution yet.

[Longdexin]

@moxcomic "同是天涯沦落人", 没错，就是最近两周出的问题，我现在基本就是手动勾选验证，先凑合着用，后面再研究研究解决办法，万恶的cloudflare！！！

[moxcomic]

@moxcomic "同是天涯沦落人", 没错，就是最近两周出的问题，我现在基本就是手动勾选验证，先凑合着用，后面再研究研究解决办法，万恶的cloudflare！！！

你这么一说倒是提醒到我了！

page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
el, err := page.Timeout(30 * time.Second).Element(`div.cf-turnstile-wrapper`)
if err != nil {
    panic(err)
}

<-time.After(time.Second * 10)

res, err := el.Eval(`() => {
const rect = this.getBoundingClientRect();
    return { x: rect.left, y: rect.top };
}`)
if err != nil {
    panic(err)
}

x, y := res.Value.Get("x").Num(), res.Value.Get("y").Num()

fmt.Println(x, y)

page.Mouse.MoveLinear(proto.NewPoint(x+27, y+29), 20)

// 使用JavaScript在鼠标位置添加一个视觉标记
page.Eval(`(x, y) => {
        const marker = document.createElement('div');
        marker.style.position = 'fixed';  // 使用fixed定位确保可见
        marker.style.left = x + 'px';
        marker.style.top = y + 'px';
        marker.style.width = '10px';
        marker.style.height = '10px';
        marker.style.backgroundColor = 'red';
        marker.style.borderRadius = '50%';
        marker.style.zIndex = 9999;  // 确保在最上层
        document.body.appendChild(marker);
    }`, x+27, y+29)

page.Mouse.MustClick(proto.InputMouseButtonLeft)
fmt.Println("click.")

我这么用能过验证（可能比手动好一点？)

[yusufmalikul]

I'm getting "Command can only be executed on top-level targets" error when PageFromTarget(proto.TargetTargetID(f.FrameID)) which f is what I got from calling el.Frame(). Wondering is this a same issue? then why the error is so different? BTW, no success with other suggested solutions (not tried old Chromium version though as I don't see it as a sustainable solution). FYI, I'm trying to create a test that needs filling credit card information in Stripe iframe form.

@alicazi did you solve the credit card filling issue? I also has the same issue when trying to access the iframe it get panic.

[lsy88]

@moxcomic“同是天涯沦落人”，正确的办法，就是最近提出的问题，我现在基本就是手动勾选验证，先凑合着用，后面再研究研究解决，万恶的cloudflare！

你这么说倒是提醒我了！

page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")
el, err := page.Timeout(30 * time.Second).Element(`div.cf-turnstile-wrapper`)
if err != nil {
    panic(err)
}

<-time.After(time.Second * 10)

res, err := el.Eval(`() => {
const rect = this.getBoundingClientRect();
    return { x: rect.left, y: rect.top };
}`)
if err != nil {
    panic(err)
}

x, y := res.Value.Get("x").Num(), res.Value.Get("y").Num()

fmt.Println(x, y)

page.Mouse.MoveLinear(proto.NewPoint(x+27, y+29), 20)

// 使用JavaScript在鼠标位置添加一个视觉标记
page.Eval(`(x, y) => {
        const marker = document.createElement('div');
        marker.style.position = 'fixed';  // 使用fixed定位确保可见
        marker.style.left = x + 'px';
        marker.style.top = y + 'px';
        marker.style.width = '10px';
        marker.style.height = '10px';
        marker.style.backgroundColor = 'red';
        marker.style.borderRadius = '50%';
        marker.style.zIndex = 9999;  // 确保在最上层
        document.body.appendChild(marker);
    }`, x+27, y+29)

page.Mouse.MustClick(proto.InputMouseButtonLeft)
fmt.Println("click.")

我用能过验证（可能比这么手动好一点？）

有新的方法解决cloudflare吗，目前尝试这种方法也不行了

[moxcomic]

@moxcomic“同是天涯沦落人”，正确的办法，就是最近提出的问题，我现在基本就是手动勾选验证，先凑合着用，后面再研究研究解决，万恶的cloudflare！

你这么说倒是提醒我了！

page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")

el, err := page.Timeout(30 * time.Second).Element(`div.cf-turnstile-wrapper`)

if err != nil {

    panic(err)

}

<-time.After(time.Second * 10)

res, err := el.Eval(`() => {

const rect = this.getBoundingClientRect();

    return { x: rect.left, y: rect.top };

}`)

if err != nil {

    panic(err)

}

x, y := res.Value.Get("x").Num(), res.Value.Get("y").Num()

fmt.Println(x, y)

page.Mouse.MoveLinear(proto.NewPoint(x+27, y+29), 20)

// 使用JavaScript在鼠标位置添加一个视觉标记

page.Eval(`(x, y) => {

        const marker = document.createElement('div');

        marker.style.position = 'fixed';  // 使用fixed定位确保可见

        marker.style.left = x + 'px';

        marker.style.top = y + 'px';

        marker.style.width = '10px';

        marker.style.height = '10px';

        marker.style.backgroundColor = 'red';

        marker.style.borderRadius = '50%';

        marker.style.zIndex = 9999;  // 确保在最上层

        document.body.appendChild(marker);

    }`, x+27, y+29)

page.Mouse.MustClick(proto.InputMouseButtonLeft)

fmt.Println("click.")

我用能过验证（可能比这么手动好一点？）

有新的方法解决cloudflare吗，目前尝试这种方法也不行了

目前我这里今天还是正常使用，应该是其他问题吧，你网站发出来看看

[lsy88]

@moxcomic“同是天涯沦落人”，正确的办法，就是最近提出的问题，我现在基本就是手动勾选验证，先凑合着用，后面再研究研究解决，万恶的cloudflare！

你这么说倒是提醒我了！

page := rod.New().ControlURL(u).MustConnect().NoDefaultDevice().MustPage("https://dash.cloudflare.com/sign-up")

el, err := page.Timeout(30 * time.Second).Element(div.cf-turnstile-wrapper)

if err != nil {

panic(err)

}

<-time.After(time.Second * 10)

res, err := el.Eval(`() => {

const rect = this.getBoundingClientRect();

return { x: rect.left, y: rect.top };

}`)

if err != nil {

panic(err)

}

x, y := res.Value.Get("x").Num(), res.Value.Get("y").Num()

fmt.Println(x, y)

page.Mouse.MoveLinear(proto.NewPoint(x+27, y+29), 20)

// 使用JavaScript在鼠标位置添加一个视觉标记

page.Eval(`(x, y) => {

    const marker = document.createElement('div');

    marker.style.position = 'fixed';  // 使用fixed定位确保可见

    marker.style.left = x + 'px';

    marker.style.top = y + 'px';

    marker.style.width = '10px';

    marker.style.height = '10px';

    marker.style.backgroundColor = 'red';

    marker.style.borderRadius = '50%';

    marker.style.zIndex = 9999;  // 确保在最上层

    document.body.appendChild(marker);

}`, x+27, y+29)

page.Mouse.MustClick(proto.InputMouseButtonLeft)

fmt.Println("click.")

我用能过验证（可能比这么手动好一点？）

有新的方法解决cloudflare吗，目前尝试这种方法也不行了

目前我这里今天还是正常使用，应该是其他问题吧，你网站发出来看看

https://nulled.to 当点击cloudflare checkbox时发现页面会同步刷新导致失效