search

berstend / puppeteer-extra

💯 Teach puppeteer new tricks through plugins.
https://extra.community
MIT License
6.55k stars 745 forks source link

[Bug] CloudFlare protection (https://nowsecure.nl) bypass on local laptop (Windows 11, display), but not on remote server (Debian + xvfb) #817

Open sngrl2 opened 1 year ago

sngrl2 commented 1 year ago

Describe the bug

I'm trying to bypass CloudFlare protection, using for test https://nowsecure.nl/ website My simple code works well on my local laptop (Windows 11 is used, with real physycal display of course), but did not work on remote server (Debian + xvfb) Mode headless: "new" is used

* This is not a problem with blacklisted IP address of the remote server: I was trying to use paid proxies and get the same result on both environments

Code Snippet

const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const {executablePath} = require('puppeteer')

puppeteer.launch({
    executablePath: executablePath(),
    headless: "new",
}).then(async browser => {
    console.log('Running tests..')
    const page = await browser.newPage()

    await page.goto('https://nowsecure.nl')
    await page.waitForTimeout(10000)

    await page.screenshot({
        path: 'stealth_test__headless-false.png',
        fullPage: true
    })
    await browser.close()
    console.log(`All done, check the screenshot`)
})

Versions, results & console output

Local:

  System:
    OS: Windows 10 10.0.22621
    CPU: (32) x64 13th Gen Intel(R) Core(TM) i9-13900HX
    Memory: 12.84 GB / 31.75 GB
  Binaries:
    Node: 18.16.0 - C:\Program Files\nodejs\node.EXE
    npm: 6.14.18 - D:\OSPanel\domains\************\node_modules\.bin\npm.CMD
  npmPackages:
    puppeteer: ^19.5.2 => 19.11.1
    puppeteer-extra: ^3.3.4 => 3.3.6
    puppeteer-extra-plugin-anonymize-ua: ^2.4.4 => 2.4.6
    puppeteer-extra-plugin-stealth: ^2.11.1 => 2.11.2
Screenshot with result: protection is passed ![stealth_test__headless-false](https://user-images.githubusercontent.com/104086284/213885018-7050e0e3-2b7d-4ed4-9aca-5033c84cb970.png)
Debug info from console ``` $ DEBUG=puppeteer-extra,puppeteer-extra-plugin:* node resources/puppeteer/puppeteer-extra-plugin-stealth.js puppeteer-extra-plugin:base:stealth Initialized. +0ms puppeteer-extra plugin registered stealth +0ms puppeteer-extra dependencies missing Set(16) { 'stealth/evasions/chrome.app', 'stealth/evasions/chrome.csi', 'stealth/evasions/chrome.loadTimes', 'stealth/evasions/chrome.runtime', 'stealth/evasions/defaultArgs', 'stealth/evasions/iframe.contentWindow', 'stealth/evasions/media.codecs', 'stealth/evasions/navigator.hardwareConcurrency', 'stealth/evasions/navigator.languages', 'stealth/evasions/navigator.permissions', 'stealth/evasions/navigator.plugins', 'stealth/evasions/navigator.webdriver', 'stealth/evasions/sourceurl', 'stealth/evasions/user-agent-override', 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions' } +3ms puppeteer-extra-plugin:base:stealth/evasions/chrome.app Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.app +5ms puppeteer-extra-plugin:base:stealth/evasions/chrome.csi Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.csi +2ms puppeteer-extra-plugin:base:stealth/evasions/chrome.loadTimes Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.loadTimes +2ms puppeteer-extra-plugin:base:stealth/evasions/chrome.runtime Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.runtime +2ms puppeteer-extra-plugin:base:stealth/evasions/defaultArgs Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/defaultArgs +1ms puppeteer-extra-plugin:base:stealth/evasions/iframe.contentWindow Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/iframe.contentWindow +2ms puppeteer-extra-plugin:base:stealth/evasions/media.codecs Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/media.codecs +2ms puppeteer-extra-plugin:base:stealth/evasions/navigator.hardwareConcurrency Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.hardwareConcurrency +2ms puppeteer-extra-plugin:base:stealth/evasions/navigator.languages Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.languages +2ms puppeteer-extra-plugin:base:stealth/evasions/navigator.permissions Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.permissions +1ms puppeteer-extra-plugin:base:stealth/evasions/navigator.plugins Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.plugins +6ms puppeteer-extra-plugin:base:stealth/evasions/navigator.webdriver Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.webdriver +2ms puppeteer-extra-plugin:base:stealth/evasions/sourceurl Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/sourceurl +2ms puppeteer-extra-plugin:base:stealth/evasions/user-agent-override Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/user-agent-override +2ms puppeteer-extra dependencies missing Set(3) { 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions', 'user-preferences' } +1ms puppeteer-extra-plugin:base:stealth/evasions/webgl.vendor Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/webgl.vendor +2ms puppeteer-extra-plugin:base:stealth/evasions/window.outerdimensions Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/window.outerdimensions +2ms puppeteer-extra-plugin:base:user-preferences Initialized. +0ms puppeteer-extra plugin registered user-preferences +3ms puppeteer-extra dependencies missing Set(1) { 'user-data-dir' } +0ms puppeteer-extra-plugin:base:user-data-dir Initialized. +0ms puppeteer-extra-plugin:user-data-dir initialized { deleteTemporary: true, deleteExisting: false, files: [], folderPath: 'C:\\Users\\Alex\\AppData\\Local\\Temp', folderPrefix: 'puppeteer_dev_profile-' } +0ms puppeteer-extra plugin registered user-data-dir +47ms puppeteer-extra ignoring dependency 'stealth/evasions/webgl.vendor', which has been required already. +0ms puppeteer-extra ignoring dependency 'stealth/evasions/window.outerdimensions', which has been required already. +0ms puppeteer-extra orderPlugins:before [ 'stealth', 'stealth/evasions/chrome.app', 'stealth/evasions/chrome.csi', 'stealth/evasions/chrome.loadTimes', 'stealth/evasions/chrome.runtime', 'stealth/evasions/defaultArgs', 'stealth/evasions/iframe.contentWindow', 'stealth/evasions/media.codecs', 'stealth/evasions/navigator.hardwareConcurrency', 'stealth/evasions/navigator.languages', 'stealth/evasions/navigator.permissions', 'stealth/evasions/navigator.plugins', 'stealth/evasions/navigator.webdriver', 'stealth/evasions/sourceurl', 'stealth/evasions/user-agent-override', 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions', 'user-preferences', 'user-data-dir' ] +1ms puppeteer-extra orderPlugins:after [ 'stealth', 'stealth/evasions/chrome.app', 'stealth/evasions/chrome.csi', 'stealth/evasions/chrome.loadTimes', 'stealth/evasions/chrome.runtime', 'stealth/evasions/media.codecs', 'stealth/evasions/navigator.hardwareConcurrency', 'stealth/evasions/navigator.languages', 'stealth/evasions/navigator.permissions', 'stealth/evasions/navigator.plugins', 'stealth/evasions/navigator.webdriver', 'stealth/evasions/sourceurl', 'stealth/evasions/user-agent-override', 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions', 'stealth/evasions/defaultArgs', 'stealth/evasions/iframe.contentWindow', 'user-preferences', 'user-data-dir' ] +1ms puppeteer-extra-plugin:user-preferences _userPrefsFromPlugins { intl: { accept_languages: 'en-US,en' } } +0ms puppeteer-extra-plugin:user-data-dir created custom dir C:\Users\Alex\AppData\Local\Temp\puppeteer_dev_profile- wnWjGk +13ms puppeteer-extra-plugin:user-data-dir Wrote file C:\Users\Alex\AppData\Local\Temp\puppeteer_dev_profile-wnWjGk\D efault\Preferences +6ms Running tests.. puppeteer-extra-plugin:stealth/evasions/user-agent-override onPageCreated - Will set these user agent options { override: { userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36', platform: 'Win32', userAgentMetadata: { brands: [Array], fullVersion: '112.0.0.0', platform: 'Windows', platformVersion: '10.0', architecture: 'x86', model: '', mobile: false }, acceptLanguage: 'en-US,en' }, opts: { userAgent: null, locale: 'en-US,en', maskLinux: true } } +0ms puppeteer-extra-plugin:user-data-dir onDisconnected +8s puppeteer-extra-plugin:user-data-dir removeUserDataDir C:\Users\Alex\AppData\Local\Temp\puppeteer_dev_profile-w nWjGk +1ms All done, check the screenshot: storage/stealth_test.png puppeteer-extra-plugin:user-data-dir null +441ms ```

Remote:

  System:
    OS: Linux 5.15 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
    CPU: (8) x64 AMD Ryzen 9 5950X 16-Core Processor
    Memory: 22.94 GB / 32.00 GB
    Container: Yes
    Shell: 5.1.4 - /bin/bash
  Binaries:
    Node: 19.3.0 - /usr/bin/node
    npm: 6.14.18 - /var/www/************/node_modules/.bin/npm
  npmPackages:
    puppeteer: ^19.5.2 => 19.11.1 
    puppeteer-extra: ^3.3.4 => 3.3.6 
    puppeteer-extra-plugin-anonymize-ua: ^2.4.4 => 2.4.6 
    puppeteer-extra-plugin-stealth: ^2.11.1 => 2.11.2
Screenshot with result: protection is NOT passed ![stealth_test__headless-true](https://user-images.githubusercontent.com/104086284/213885470-b40f5202-35d8-4e51-b779-54421f6d7b75.png)
Debug info from console ``` # sudo -u www-data DEBUG=puppeteer-extra,puppeteer-extra-plugin:* node resources/puppeteer/puppeteer-extra-plugin-stealth.js puppeteer-extra-plugin:base:stealth Initialized. +0ms puppeteer-extra plugin registered stealth +0ms puppeteer-extra dependencies missing Set(16) { 'stealth/evasions/chrome.app', 'stealth/evasions/chrome.csi', 'stealth/evasions/chrome.loadTimes', 'stealth/evasions/chrome.runtime', 'stealth/evasions/defaultArgs', 'stealth/evasions/iframe.contentWindow', 'stealth/evasions/media.codecs', 'stealth/evasions/navigator.hardwareConcurrency', 'stealth/evasions/navigator.languages', 'stealth/evasions/navigator.permissions', 'stealth/evasions/navigator.plugins', 'stealth/evasions/navigator.webdriver', 'stealth/evasions/sourceurl', 'stealth/evasions/user-agent-override', 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions' } +1ms puppeteer-extra-plugin:base:stealth/evasions/chrome.app Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.app +2ms puppeteer-extra-plugin:base:stealth/evasions/chrome.csi Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.csi +0ms puppeteer-extra-plugin:base:stealth/evasions/chrome.loadTimes Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.loadTimes +1ms puppeteer-extra-plugin:base:stealth/evasions/chrome.runtime Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/chrome.runtime +1ms puppeteer-extra-plugin:base:stealth/evasions/defaultArgs Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/defaultArgs +0ms puppeteer-extra-plugin:base:stealth/evasions/iframe.contentWindow Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/iframe.contentWindow +0ms puppeteer-extra-plugin:base:stealth/evasions/media.codecs Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/media.codecs +1ms puppeteer-extra-plugin:base:stealth/evasions/navigator.hardwareConcurrency Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.hardwareConcurrency +0ms puppeteer-extra-plugin:base:stealth/evasions/navigator.languages Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.languages +1ms puppeteer-extra-plugin:base:stealth/evasions/navigator.permissions Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.permissions +0ms puppeteer-extra-plugin:base:stealth/evasions/navigator.plugins Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.plugins +1ms puppeteer-extra-plugin:base:stealth/evasions/navigator.webdriver Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/navigator.webdriver +1ms puppeteer-extra-plugin:base:stealth/evasions/sourceurl Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/sourceurl +0ms puppeteer-extra-plugin:base:stealth/evasions/user-agent-override Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/user-agent-override +1ms puppeteer-extra dependencies missing Set(3) { 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions', 'user-preferences' } +0ms puppeteer-extra-plugin:base:stealth/evasions/webgl.vendor Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/webgl.vendor +0ms puppeteer-extra-plugin:base:stealth/evasions/window.outerdimensions Initialized. +0ms puppeteer-extra plugin registered stealth/evasions/window.outerdimensions +1ms puppeteer-extra-plugin:base:user-preferences Initialized. +0ms puppeteer-extra plugin registered user-preferences +1ms puppeteer-extra dependencies missing Set(1) { 'user-data-dir' } +0ms puppeteer-extra-plugin:base:user-data-dir Initialized. +0ms puppeteer-extra-plugin:user-data-dir initialized { deleteTemporary: true, deleteExisting: false, files: [], folderPath: '/tmp', folderPrefix: 'puppeteer_dev_profile-' } +0ms puppeteer-extra plugin registered user-data-dir +16ms puppeteer-extra ignoring dependency 'stealth/evasions/webgl.vendor', which has been required already. +0ms puppeteer-extra ignoring dependency 'stealth/evasions/window.outerdimensions', which has been required already. +0ms puppeteer-extra orderPlugins:before [ 'stealth', 'stealth/evasions/chrome.app', 'stealth/evasions/chrome.csi', 'stealth/evasions/chrome.loadTimes', 'stealth/evasions/chrome.runtime', 'stealth/evasions/iframe.contentWindow', 'stealth/evasions/media.codecs', 'stealth/evasions/navigator.hardwareConcurrency', 'stealth/evasions/navigator.languages', 'stealth/evasions/navigator.permissions', 'stealth/evasions/navigator.plugins', 'stealth/evasions/navigator.webdriver', 'stealth/evasions/sourceurl', 'stealth/evasions/user-agent-override', 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions', 'user-preferences', 'user-data-dir' ] +0ms puppeteer-extra orderPlugins:after [ 'stealth', 'stealth/evasions/chrome.app', 'stealth/evasions/chrome.csi', 'stealth/evasions/chrome.loadTimes', 'stealth/evasions/chrome.runtime', 'stealth/evasions/media.codecs', 'stealth/evasions/navigator.hardwareConcurrency', 'stealth/evasions/navigator.languages', 'stealth/evasions/navigator.permissions', 'stealth/evasions/navigator.plugins', 'stealth/evasions/navigator.webdriver', 'stealth/evasions/sourceurl', 'stealth/evasions/user-agent-override', 'stealth/evasions/webgl.vendor', 'stealth/evasions/window.outerdimensions', 'stealth/evasions/defaultArgs', 'stealth/evasions/iframe.contentWindow', 'user-preferences', 'user-data-dir' ] +0ms puppeteer-extra-plugin:user-preferences _userPrefsFromPlugins { intl: { accept_languages: 'en-US,en' } } +0ms puppeteer-extra-plugin:user-data-dir created custom dir /tmp/puppeteer_dev_profile-DmMi0c +4ms puppeteer-extra-plugin:user-data-dir Wrote file /tmp/puppeteer_dev_profile-DmMi0c/Default/Preferences +5ms Running tests.. puppeteer-extra-plugin:stealth/evasions/user-agent-override onPageCreated - Will set these user agent options { override: { userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36', platform: 'Win32', userAgentMetadata: { brands: [Array], fullVersion: '112.0.0.0', platform: 'Windows', platformVersion: '10.0', architecture: 'x86', model: '', mobile: false }, acceptLanguage: 'en-US,en' }, opts: { userAgent: null, locale: 'en-US,en', maskLinux: true } } +0ms puppeteer-extra-plugin:user-data-dir onDisconnected +6s puppeteer-extra-plugin:user-data-dir removeUserDataDir /tmp/puppeteer_dev_profile-DmMi0c +0ms All done, check the screenshot: storage/stealth_test.png puppeteer-extra-plugin:user-data-dir null +212ms ```
Roncarlos commented 1 year ago

I have the exact same prob, did you find something that works?

lastelok commented 1 year ago

same problem

Roncarlos commented 1 year ago

The only diff i see on https://bot.sannysoft.com/ is my media devices that are not existent in debian but exists in Windows: In windows image In debian image

agallio commented 1 year ago

Can you try to add --auto-open-devtools-for-tabs to your puppeteer.launch args. Something like this:

puppeteer.launch({
  args: ['--auto-open-devtools-for-tabs']
})

I tried and it works. I got this solution from here https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1390

Roncarlos commented 1 year ago

js args: [ "--disable-notifications", "--auto-open-devtools-for-tabs", "--no-sandbox", "--window-size=1280,720", "--disable-dev-shm-usage" ],

I'm already using it, i tried with disable-dev-shm-usage too, but not working either. But still works on windows.

Roncarlos commented 1 year ago

I found why, i was creating a new browser and then a new tab where i go to the protected link. By reusing the default about:blank page it's working. let page = (await browser.pages())[0]; // Instead of using browser.newPage()

GrayR0ot commented 1 year ago

Same problem, but in my case windows 11 not works too:

const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const {executablePath} = require('puppeteer')

puppeteer.launch({
    executablePath: executablePath(),
    headless: "new",
    args: [
        '--no-sandbox',
        '--auto-open-devtools-for-tabs',
        '--disable-notifications',
        '--disable-dev-shm-usage',
    ],
}).then(async browser => {
    console.log('Running tests..')
    const page = await browser.newPage()

    await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5643.204 Safari/537.36')
    await page.goto('https://nowsecure.nl')
    await page.waitForTimeout(10000)

    await page.screenshot({
        path: 'stealth_test__headless-false.png',
        fullPage: true
    })
    await browser.close()
    console.log(`All done, check the screenshot`)
})
luluhoc commented 1 year ago

Devtools solution was patched

thanh858036 commented 1 year ago

cloudflare update fix devtools

GrayR0ot commented 1 year ago

So, actually there is nothing that can bypass UAM ?

luluhoc commented 1 year ago

So, actually there is nothing that can bypass UAM ?

Right

jgsaez9 commented 1 year ago

same problem

jgsaez9 commented 1 year ago

I have discovered that if you specify your current user agent, Cloudflare lets you pass without any issues. If you change any value of it, it won't let you through.

This rules out the theory that it could be the IP and also indicates that there is some kind of fingerprinting that Cloudflare is using to compare our user agent with the original.

Any modification to the user agent will be detected by Cloudflare.

This code (adding your team's user agent) will pass just fine.

const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const {executablePath} = require('puppeteer')

puppeteer.launch({
    executablePath: executablePath(),
    headless: "new",
    args: [
        '--no-sandbox',
        '--auto-open-devtools-for-tabs',
        '--disable-notifications',
        '--disable-dev-shm-usage',
    ],
}).then(async browser => {
    console.log('Running tests..')
    const page = await browser.newPage()

    await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36')
    await page.goto('https://nowsecure.nl')
    await page.waitForTimeout(10000)

    await page.screenshot({
        path: 'stealth_test__headless-false.png',
        fullPage: true
    })
    await browser.close()
    console.log(`All done, check the screenshot`)
})
luluhoc commented 1 year ago

I have discovered that if you specify your current user agent, Cloudflare lets you pass without any issues. If you change any value of it, it won't let you through.

This rules out the theory that it could be the IP and also indicates that there is some kind of fingerprinting that Cloudflare is using to compare our user agent with the original.

Any modification to the user agent will be detected by Cloudflare.

This code (adding your team's user agent) will pass just fine.


const puppeteer = require('puppeteer-extra')

const StealthPlugin = require('puppeteer-extra-plugin-stealth')

puppeteer.use(StealthPlugin())

const {executablePath} = require('puppeteer')

puppeteer.launch({

    executablePath: executablePath(),

    headless: "new",

    args: [

        '--no-sandbox',

        '--auto-open-devtools-for-tabs',

        '--disable-notifications',

        '--disable-dev-shm-usage',

    ],

}).then(async browser => {

    console.log('Running tests..')

    const page = await browser.newPage()

    await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36')

    await page.goto('https://nowsecure.nl')

    await page.waitForTimeout(10000)

    await page.screenshot({

        path: 'stealth_test__headless-false.png',

        fullPage: true

    })

    await browser.close()

    console.log(`All done, check the screenshot`)

})

I don’t think it works I tried it today with the same UA as browser and still got blocked.

jgsaez9 commented 1 year ago

I have discovered that if you specify your current user agent, Cloudflare lets you pass without any issues. If you change any value of it, it won't let you through. This rules out the theory that it could be the IP and also indicates that there is some kind of fingerprinting that Cloudflare is using to compare our user agent with the original. Any modification to the user agent will be detected by Cloudflare. This code (adding your team's user agent) will pass just fine.


const puppeteer = require('puppeteer-extra')

const StealthPlugin = require('puppeteer-extra-plugin-stealth')

puppeteer.use(StealthPlugin())

const {executablePath} = require('puppeteer')

puppeteer.launch({

    executablePath: executablePath(),

    headless: "new",

    args: [

        '--no-sandbox',

        '--auto-open-devtools-for-tabs',

        '--disable-notifications',

        '--disable-dev-shm-usage',

    ],

}).then(async browser => {

    console.log('Running tests..')

    const page = await browser.newPage()

    await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36')

    await page.goto('https://nowsecure.nl')

    await page.waitForTimeout(10000)

    await page.screenshot({

        path: 'stealth_test__headless-false.png',

        fullPage: true

    })

    await browser.close()

    console.log(`All done, check the screenshot`)

})

I don’t think it works I tried it today with the same UA as browser and still got blocked.

With the latest version of Google Chrome updated on your computer?

For it to work, you must have the latest version of Google Chrome up to date and set the same useragent as the one on your computer

Av4aR commented 1 year ago

Even though this is definitely not a great solution - I've found a workaround to get past Cloudflare for now.

They somehow detect if the browser is controlled by an automation script. So, the simplest approach would be to just.. not do that. If we manually launch our browser with the --remote-debugging-port=PORT flag and connect to it via Puppeteer.connect we can Page.goto to our site of choice and if we call Browser.disconnect immediatly after Page.goto Cloudflare will let us through, as the browser is no longer controlled by Puppeteer.

After that we just need to confirm we actually got past Cloudflare (there are probably multiple better ways to accomplish this, I just open another headless browser and pull some data from http://127.0.0.1:PORT/json/list to see if it matches the site I'm trying to access). Then we can reconnect to our browser again using Puppeteer.connect and use Puppeteer like we normally would.

You could also inject javascript into the page before you disconnect using Page.evaluateOnNewDocument.

This probably doesn't work well for all usecases, but for what I'm doing it's okay. //Edit: Forgot to mention, I'm doing this with headful chromium.

putuoka commented 1 year ago

Hey everyone, I believe this thing might be useful for you. https://github.com/CheshireCaat/puppeteer-with-fingerprints

devblack commented 1 year ago 
((element, selector, { pQuerySelector }) => {
        return pQuerySelector(element, selector);
    }
//# 

sourceURL=pptr:evaluateHandle;

CDPElementHandle.evaluateHandle (D:\......\node_modules\puppeteer-core\lib\cjs\puppeteer\api\ElementHandle.js:101:28)
)

Cloudflare is detecting puppeteer.

jozsi commented 1 year ago 
((element, selector, { pQuerySelector }) => {
        return pQuerySelector(element, selector);
    }
//# 

sourceURL=pptr:evaluateHandle;

CDPElementHandle.evaluateHandle (D:\......\node_modules\puppeteer-core\lib\cjs\puppeteer\api\ElementHandle.js:101:28)
)

Cloudflare is detecting puppeteer.

Something indeed has happened since yesterday, cause if I click "I'm a human", the Turnstile "captcha" doesn't appear anymore, it just redirects and asks to click the checkbox again, and again.

GigaBigBoss commented 1 year ago

Hey everyone, I believe this thing might be useful for you. https://github.com/CheshireCaat/puppeteer-with-fingerprints

Your paid package doesn't work, but nice attempt at marketing I guess.

fuzeman commented 1 year ago

It took some time to identify this issue, but I finally found a simple fix without any changes to Puppeteer.

Just add the following to puppeteer.connect or puppeteer.launch options:

targetFilter: (target) => !!target.url()

This does prevent puppeteer from clicking specific elements inside the captcha iframe, but directly clicking the iframe to pass the captcha works perfectly fine in my tests.

luluhoc commented 1 year ago

@fuzeman I'm getting error on init target.url() is not a function

fuzeman commented 1 year ago

@luluhoc Looks like targetFilter was changed in puppeteer v21 (https://github.com/puppeteer/puppeteer/commit/44712d1e6efcb3fa49c27b1195d17c0c1c92a0ca).

I think this will work with older versions of puppeteer (haven't tested myself):

targetFilter: (target) => !!target.url
luluhoc commented 1 year ago

@luluhoc Looks like targetFilter was changed in puppeteer v21 (https://github.com/puppeteer/puppeteer/commit/44712d1e6efcb3fa49c27b1195d17c0c1c92a0ca).

I think this will work with older versions of puppeteer (haven't tested myself):


targetFilter: (target) => !!target.url

@fuzeman the solution doesn't give me error anymore but it doesn't allow me to pass CF

fuzeman commented 1 year ago

@luluhoc Have you previously been passing Cloudflare Protection (in the last month)?

There are many other variables that could impact passing bot detection. I have typically run puppeteer in non-headless mode with the stealth and humanize plugins in puppeteer-extra, but some stealth evasions need to be disabled to avoid making you more detectable as a bot.

I have also had more success recently using GoLogin which can generate random browser fingerprints with many bot detection evasion features built-in.

thanh858036 commented 1 year ago

@fuzeman

targetFilter: (target) => !!target.url()

can not switch frame to click captcha checkbox

fuzeman commented 1 year ago

@thanh858036 you need to click the captcha iframe using raw mouse events - either with a patched version of the humanize plugin (which is broken, and has been removed from the repository), or using puppeteer.mouse yourself.

Motivationking commented 1 year ago

targetFilter: (target) => !!target.url

this is worked for me... I found a lot of ways but it wont work, now it tried this one and IT IS PERFECTLY WORKED...

ppyrzanowski commented 1 year ago

I can conclude from testing that Cloudflare is able to detect whether dev-tools are open or not. I was able to manually pass the cf captcha by commenting out the launcher argument --auto-open-devtools-for-tabs and setting headless to false. With dev-tools open, i always was stuck in the captach reload loop no matter if the browser was instrumented or not...

putuoka commented 1 year ago

AI said https://www.browserling.com/ & brow.sh can bypass https://nowsecure.nl

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://browser.brow.sh:4444?token=YOUR_API_TOKEN` 
});
GrayR0ot commented 1 year ago

browsh is self hosted right ?

Motivationking commented 1 year ago

AI said https://www.browserling.com/ & brow.sh can bypass https://nowsecure.nl

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://browser.brow.sh:4444?token=YOUR_API_TOKEN` 
});

is it worked with puppeteer https://www.browserling.com/

ffastym commented 1 year ago

the same issue when trying to scrape www.indeed.com. Any updates?

Linzh1998 commented 1 year ago

I also passed Windows, but failed Linux 


const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const {executablePath} = require('puppeteer')

puppeteer.launch({
    // args: ['--auto-open-devtools-for-tabs'],
    args: ['--no-sandbox'],
//    headless: true,
    executablePath: executablePath(),
    targetFilter: (target) => !!target.url(), // 使用 targetFilter 过滤目标
    headless: "new",
    // 设置日志级别为 'info' 或 'debug'
    logLevel: 'debug', // 或 'debug'
}).then(async browser => {
    console.log('Running tests..')
//    const page = await browser.newPage()
    let page = (await browser.pages())[0]; // Instead of using browser.newPage()
//        await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36')

//    await page.goto('https://nowsecure.nl')
    try {
        await page.goto('https://nowsecure.nl')
    } catch (error) {
        console.error('Failed to navigate:', error)
    }

    await page.waitForTimeout(10000)

    //add
    //await page.waitForTimeout(5000)

    // 点击页面空白内容
    await page.click('body')

    await page.waitForTimeout(500)

    // 模拟按 Tab 键多次，直到焦点达到复选框
    await page.keyboard.press('Tab')
    // await page.keyboard.press('Tab')

    await page.waitForTimeout(500)

    // 模拟按 Space 键选择复选框
    await page.keyboard.press('Space')

    // 等待一段时间，可以根据需要调整
    await page.waitForTimeout(15000)
    //end

    await page.screenshot({
        path: 'stealth_test__headless-false.png',
        fullPage: true
    })
    // await browser.close()
    console.log(`All done, check the screenshot`)
})
sngrl2 commented 1 year ago

@berstend more than 3 months without working solution, can you help us please?

sngrl2 commented 1 year ago

@agallio

Can you try to add --auto-open-devtools-for-tabs to your puppeteer.launch args. Something like this:

puppeteer.launch({
  args: ['--auto-open-devtools-for-tabs']
})

I tried and it works. I got this solution from here ultrafunkamsterdam/undetected-chromedriver#1390

Does not work right now, Cloudflare detects devtools and show the unsolvable captcha

sngrl2 commented 1 year ago

@Roncarlos

I found why, i was creating a new browser and then a new tab where i go to the protected link. By reusing the default about:blank page it's working. let page = (await browser.pages())[0]; // Instead of using browser.newPage()

Nothing changed: not better, nor worst

sngrl2 commented 1 year ago

@fuzeman

It took some time to identify this issue, but I finally found a simple fix without any changes to Puppeteer.

Just add the following to puppeteer.connect or puppeteer.launch options:

targetFilter: (target) => !!target.url()

This does prevent puppeteer from clicking specific elements inside the captcha iframe, but directly clicking the iframe to pass the captcha works perfectly fine in my tests.

After 3 months Cloudflare started to show captcha on my local laptop, and your snippet helps me to solve it 🥳 But not on remote server with Debian & xvfb

sngrl2 commented 1 year ago

@Linzh1998 hmm, it's seems like your solution for manual solving of the captcha is working in combination with other advises from this thread

This is my modified snippet which allow to pass Cloudflare protection and solve the captcha on the local laptop (Windows) and remote server (Debian & xvfb) at this moment:

/**
 * @see https://github.com/berstend/puppeteer-extra/issues/817
 */
const puppeteer = require('puppeteer-extra')
const {executablePath} = require('puppeteer')

const StealthPlugin = require('puppeteer-extra-plugin-stealth')
const stealth = StealthPlugin()
puppeteer.use(stealth)

let resultImagePath = 'storage/stealth_test.png'

puppeteer.launch({
    executablePath: executablePath(),
    readTimeout: 5 * 60 * 1000,
    headless: 'new',
    targetFilter: (target) => !!target.url,
}).then(async browser => {
    console.log('Running tests..')
    let page = (await browser.pages())[0]; // Instead of using browser.newPage()

    await page.goto('https://nowsecure.nl')
    await page.waitForTimeout(5000)

    // try to solve the captcha manually
    console.log(`Try to solve the captcha manually...`)

    await page.click('body')
    await page.waitForTimeout(500)

    await page.keyboard.press('Tab')
    await page.waitForTimeout(500)

    await page.keyboard.press('Space')
    await page.waitForTimeout(10000)
    // finish

    await page.screenshot({
        path: resultImagePath,
        fullPage: true
    })
    await browser.close()
    console.log(`All done, check the screenshot: ` + resultImagePath)
})
atalantus commented 1 year ago

This does not work for me locally (Windows). I tried the newest puppeteer version and multiple different configurations which helped previously. Seems to me like Cloudflare more or less recently found another way of detecting puppeteer.

atalantus commented 1 year ago

At least for me, your site detects my puppeteer configurations quite well. Nice job.

atalantus commented 1 year ago

For me, it did not detect my Chrome, neither normal nor incognito.

AngelliaX commented 11 months ago

any news guys

mdervisaygan commented 11 months ago

Hello, https://github.com/zfcsoftware/youtube_lessons_resources/blob/main/cloudflare_bypass/index.js You can get cookie and agent with cloudflare scraper like here and send requests with that information. If you need a proxy: https://github.com/zfcsoftware/puppeteer-real-browser you can use this.

AHgPuK commented 10 months ago

@sngrl2 I found that headless: false passes just fine. If I set headless: 'new' it stops to work even on Windows. So, I think there is a difference between headless and non-headless modes.

KaKi87 commented 9 months ago 
targetFilter: (target) => !!target.url()

How to make this work with browser.newPage() ?

Thanks

andrii-shpontak commented 9 months ago

Also faced this problem...

KaKi87 commented 9 months ago 
await page.evaluate(`window.open('https://nowsecure.nl')`)

This one opens a new tab but never loads https://nowsecure.nl and gets stuck on about:blank instead ;

await page.evaluate(`window.open('https://nowsecure.nl', '_self')`)

This one opens the URL in the current tab.

There was a bug in cloudflare itself, they finally resolved it.

I'm still experiencing this issue...

KaKi87 commented 9 months ago

force it to be solved

What do you mean ?

AHgPuK commented 9 months ago

It is a CDP detection in iframe :(

Can you explain why it does work in non-headless mode?