[Bug] Stealth plugin's iframe.contentWindow deletes the entire DOM and crashes site in an unrecoverable error

AbraarArique commented 3 months ago

Describe the bug

When visiting certain pages (see full code example below), puppeteer-extra-plugin-stealth's iframe.contentWindow evasion interacts with the site's JavaScript to cause the entire DOM/HTML to go blank or get deleted.

No direct errors are thrown, which made it very difficult to pinpoint the cause of this issue when I first encountered it.

But after tweaking many Puppeteer settings, I found that puppeteer-extra-plugin-stealth is causing it.

Then I was able to track it down to a specific evasion: iframe.contentWindow.

Code Snippet

import pptr from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';

const puppeteer = pptr.default;

const evasions = new Set([
  'chrome.app',
  'chrome.csi',
  'chrome.loadTimes',
  'chrome.runtime',
  'defaultArgs',
  'iframe.contentWindow',
  'media.codecs',
  'navigator.hardwareConcurrency',
  'navigator.languages',
  'navigator.permissions',
  'navigator.plugins',
  'navigator.webdriver',
  'sourceurl',
  'user-agent-override',
  'webgl.vendor',
  'window.outerdimensions',
]);
puppeteer.use(StealthPlugin({ enabledEvasions: evasions }));

const browser = await puppeteer.launch();
const page = await browser.newPage();

// This try/catch is needed because this site will often exceed the default 30s timeout
try {
  await page.goto(
    'https://variety.com/2024/film/global/johnny-depp-modi-premiere-san-sebastian-film-festival-1236111999/',
  );
} catch {}

const body = await page.evaluate(() => document.body.outerHTML.slice(0, 50));
console.log(body);

When I run this code, this error is thrown:

node:internal/process/esm_loader:34
      internalBinding('errors').triggerUncaughtException(
                                ^

Error [TypeError]: Cannot read properties of null (reading 'outerHTML')
    at evaluate (evaluate at file:///Users/abraar/Documents/monorepo/packages/ai-dataset-bot/src/run.ts:61:1242, <anonymous>:0:19)
    at ExecutionContext.#evaluate (/Users/abraar/Documents/monorepo/node_modules/.pnpm/puppeteer-core@22.6.5/node_modules/puppeteer-core/src/cdp/ExecutionContext.ts:304:34)
    at ExecutionContext.evaluate (/Users/abraar/Documents/monorepo/node_modules/.pnpm/puppeteer-core@22.6.5/node_modules/puppeteer-core/src/cdp/ExecutionContext.ts:157:12)
    at IsolatedWorld.evaluate (/Users/abraar/Documents/monorepo/node_modules/.pnpm/puppeteer-core@22.6.5/node_modules/puppeteer-core/src/cdp/IsolatedWorld.ts:143:12)
    at CdpFrame.evaluate (/Users/abraar/Documents/monorepo/node_modules/.pnpm/puppeteer-core@22.6.5/node_modules/puppeteer-core/src/api/Frame.ts:470:12)
    at CdpPage.evaluate (/Users/abraar/Documents/monorepo/node_modules/.pnpm/puppeteer-core@22.6.5/node_modules/puppeteer-core/src/api/Page.ts:2190:12)
    at <anonymous> (/Users/abraar/Documents/monorepo/packages/ai-dataset-bot/src/run.ts:488:14)

Node.js v20.11.1

The reason why the TypeError occurs is because document.body is null.

In fact, if you turn off headless and inspect visually, you'll first see the site load normally, but then everything on the page goes blank, and all HTML elements inside Chrome DevTools have disappeared.

But if you comment out iframe.contentWindow from the list of evasions, it works properly:

<body class="home blog pmc-gallery__ pmc-desktop p

For the particular site above, I also discovered that if you block all requests that include the path pmc-plugins in DevTools (this is a WordPress plugin), the above error doesn't occur.

So I'm guessing that stealth plugin's iframe.contentWindow somehow messes up this site's internal JS scripts.

This webpage also seems to contain <iframe> tags with srcdoc property, which may be relevant to this issue.

I've also experienced this issue on other sites run by the same organization, such as https://www.billboard.com/

Versions

  System:
    OS: macOS 12.7.4
    CPU: (4) x64 Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
    Memory: 786.00 MB / 8.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 20.11.1 - /usr/local/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 10.2.4 - /usr/local/bin/npm
    pnpm: 9.5.0 - /usr/local/bin/pnpm
  npmPackages:
    puppeteer-core: ^22.6.5 => 22.6.5 
    puppeteer-extra: ^3.3.6 => 3.3.6 
    puppeteer-extra-plugin-adblocker: ^2.13.6 => 2.13.6 
    puppeteer-extra-plugin-stealth: ^2.11.2 => 2.11.2

dannyokec commented 1 month ago

Have you been able to resolve this or make we enter the matter ?

vladtreny commented 1 month ago

just remove iframe.contentWindow. This works:

import puppeteer from 'puppeteer-extra'
import pptr from 'puppeteer-extra'

            const puppeteer = pptr.default

            const evasions = new Set([
                'chrome.app',
                'chrome.csi',
                'chrome.loadTimes',
                'chrome.runtime',
                'defaultArgs',
                //  'iframe.contentWindow',
                'media.codecs',
                'navigator.hardwareConcurrency',
                'navigator.languages',
                'navigator.permissions',
                'navigator.plugins',
                'navigator.webdriver',
                'sourceurl',
                'user-agent-override',
                'webgl.vendor',
                'window.outerdimensions'
            ])
            console.log('here')
            puppeteer.use(StealthPlugin({enabledEvasions: evasions}))

            const browser = await puppeteer.launch({
                userDataDir: '_',
                executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',

                headless: false  // <--- 1
            })
            const page = await browser.newPage()

// This try/catch is needed because this site will often exceed the default 30s timeout
            try {
                await page.goto(
                    'https://variety.com/2024/film/global/johnny-depp-modi-premiere-san-sebastian-film-festival-1236111999/', {timeout: 5_000}
                )
            } catch {}

            const body = await page.evaluate(() => document.body.outerHTML.slice(0, 50))
            console.log(body)

berstend / puppeteer-extra

[Bug] Stealth plugin's iframe.contentWindow deletes the entire DOM and crashes site in an unrecoverable error #909