avoid bot detection - Githubissues

steamboatid commented 1 year ago

hi,

is there any way to avoid bot detection like https://bot.sannysoft.com/ ?

thx.

ayd-is commented 1 year ago

$chromeOptions = ['--disable-blink-features=AutomationControlled'];
/// - webdrive

$browser = $browserFactory->createBrowser([
'customFlags' => $chromeOptions
]);

$plugins = [
        [
            'name' => 'PDF Viewer',
            'description' => 'Portable Document Format',
            'filename' => 'internal-pdf-viewer',
            'mimeTypes' => [
                'application/pdf~pdf~Portable Document Format',
                'text/pdf~pdf~Portable Document Format'
            ]
        ],
        [
            'name' => 'Chrome PDF Viewer',
            'description' => 'Portable Document Format',
            'filename' => 'internal-pdf-viewer',
            'mimeTypes' => [
                'application/pdf~pdf~Portable Document Format',
                'text/pdf~pdf~Portable Document Format'
            ]
        ],
        [
            'name' => 'Chromium PDF Viewer',
            'description' => 'Portable Document Format',
            'filename' => 'internal-pdf-viewer',
            'mimeTypes' => [
                'application/pdf~pdf~Portable Document Format',
                'text/pdf~pdf~Portable Document Format'
            ]
        ],
        [
            'name' => 'Microsoft Edge PDF Viewer',
            'description' => 'Portable Document Format',
            'filename' => 'internal-pdf-viewer',
            'mimeTypes' => [
                'application/pdf~pdf~Portable Document Format',
                'text/pdf~pdf~Portable Document Format'
            ]
        ],
        [
            'name' => 'WebKit built-in PDF',
            'description' => 'Portable Document Format',
            'filename' => 'internal-pdf-viewer',
            'mimeTypes' => [
                'application/pdf~pdf~Portable Document Format',
                'text/pdf~pdf~Portable Document Format'
            ]
        ]
    ];

    $mimetypes = [
    ['Portable Document Format~~application/pdf~~pdf'],
    ['Portable Document Format~~text/pdf~~pdf']
     ];

         $fakePlugins = json_encode($plugins);
         $fakemimetypess = json_encode($mimetypes);
         $fakeLanguages = json_encode(['ru-RU', 'ru', 'en-US', 'en']);

          $fakePlugins = addslashes($fakePlugins);
          $fakemimetypess = addslashes($fakemimetypess);
          $fakeLanguages = addslashes($fakeLanguages);

          // $fakePlugins = "{\"0\":{\"0\":{},\"1\":{}},\"1\":{\"0\":{},\"1\":{}},\"2\":{\"0\":{},\"1\":{}},\"3\":{\"0\":{},\"1\":{}},\"4\":{\"0\":{},\"1\":{}}}";
          // {"0":{"0":{},"1":{}},"1":{"0":{},"1":{}},"2":{"0":{},"1":{}},"3":{"0":{},"1": {}},"4":{"0":{},"1":{}}}
          // {"0":{"0":{},"1":{}},"1":{"0":{},"1":{}},"2":{"0":{},"1":{}},"3":{"0":{},"1":{}},"4":{"0":{},"1":{}}}
          // 

        $js_code = <<<JSCODE
        // кома или не комп или вид компа
        Object.defineProperty(navigator, 'platform', {
        get: () => 'MacIntel',
        });

        // Устанавливаем фейковые плагины и языки
        Object.defineProperty(navigator, 'plugins', {
        get: () => JSON.parse('$fakePlugins'),
        });

        // Устанавливаем фейковые плагины и языки
        Object.defineProperty(navigator, 'mimeTypes', {
        get: () => JSON.parse('$fakemimetypess'),
        });

        // Добавляем фейковый объект chrome
        Object.defineProperty(window, 'chrome', {
        get: () => ({
        runtime: {},
        webstore: {},
        app: {},
        csi: {},
        loadTimes: {},
        connect: {},
        // Добавьте другие свойства, если нужно
        }),
        });

        Object.defineProperty(navigator, 'languages', {
        get: () => JSON.parse('$fakeLanguages'),
        });

        // Симулируем разрешения навигатора
        const originalQuery = window.navigator.permissions.query;
        window.navigator.permissions.query = (parameters) => (
        parameters.name === 'notifications' ?
        Promise.resolve({ state: Notification.permission }) :
        originalQuery(parameters)
        );

        const getParameter = WebGLRenderingContext.getParameter;
        WebGLRenderingContext.prototype.getParameter = function(parameter) {
        // UNMASKED_VENDOR_WEBGL
        if (parameter === 37445) {
        return 'Google Inc. (Intel Inc.)';
        }
        // UNMASKED_RENDERER_WEBGL
        if (parameter === 37446) {
        return 'ANGLE (Intel Inc., Intel(R) Iris(TM) Plus Graphics OpenGL Engine, OpenGL 4.1)';
        }

        return getParameter(parameter);
        };

        // store the existing descriptor
        const elementDescriptor = Object.getOwnPropertyDescriptor(HTMLElement.prototype, 'offsetHeight');

        // redefine the property with a patched descriptor
        Object.defineProperty(HTMLDivElement.prototype, 'offsetHeight', {
        ...elementDescriptor,
        get: function() {
        if (this.id === 'modernizr') {
        return 1;
        }
        return elementDescriptor.get.apply(this);
        },
        });
        JSCODE;

        // // Симулируем разрешения навигатора
        // const originalQuery = window.navigator.permissions.query;
        // window.navigator.permissions.query = (parameters) => (
        // parameters.name === 'notifications' ?
        // Promise.resolve({ state: Notification.permission }) :
        // originalQuery(parameters)
        // );
        // 

          $browser->setPagePreScript($js_code);

            $page = $browser->createPage();

            //  $page = $browser->createPage();

              $navigation = $page->navigate('https://bot.sannysoft.com/');

device - MacBook

quyunet commented 1 year ago

define('USERAGENT', 'Mozilla/5.0 (X11; Debian; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36');
$options = [
    'userAgent' => USERAGENT,
    'headless' => false,
    'keepAlive' => true,
    'enableImages' => true,
    'startupTimeout' => 60,
    'noSandbox' => false,
    'ignoreCertificateErrors' => true,
    'customFlags' => ['--lang=en-US','--disable-blink-features=AutomationControlled'],
];

//$factory = new \HeadlessChromium\BrowserFactory('google-chrome-stable');    //Linux
$factory = new \HeadlessChromium\BrowserFactory('C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe');    //Windows
$browser = $factory->createBrowser($options);

$browser->setPagePreScript("Simulate navigator permissions;
const originalQuery = window.navigator.permissions.query;
Object.defineProperty(navigator, 'webdriver', {get: () => undefined,});
window.navigator.permissions.query = (parameters) => (
    parameters.name === 'notifications' ?
        Promise.resolve({ state: Notification.permission }) :
        originalQuery(parameters)
);");
// do something with the browser
$page = $browser->createPage();

$url = 'https://bot.sannysoft.com/';

$page->setUserAgent(USERAGENT);

$page->navigate($url)->waitForNavigation('load');

$browser->close();

although all passes are possible, some websites still cannot pass the detection, suggested reference: https://github.com/ultrafunkamsterdam/undetected-chromedriver https://github.com/berstend/puppeteer-extra/tree/master/packages

GrahamCampbell commented 1 year ago

I'm going to close this, since use of this library is a bot.

chrome-php / chrome

avoid bot detection #500