[Bug] bet365.com stopped working 2 days ago

[apulidoc]

edit: Further discussion should happen on the community discord

---

Describe the bug

• Im trying to scrap bet365 website
• Expect open the website bet365.com without problems.
• I was using puppeteer-extra-plugin-stealth for this purpose but is not working anymore from 2 days ago.

Code Snippet

const fs = require('fs')
const json = require('JSON')
const puppeteer = require('puppeteer-extra')
const { Bot } = require('tgapi')
const mysql = require('mysql')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')

var dbConfig ={
  host     : 'localhost',
  port     : '3307',
  user     : 'xxxx',
  password : 'xxxx',
  database : 'mydb'
}

let connection = mysql.createPool(dbConfig);
connection.setMaxListeners(20)

puppeteer.use(StealthPlugin())

//Ejecucion de Puppeteer
puppeteer.launch({ args: ['--start-maximized'], headless: false }).then(async browser => {
  console.log('Ejecutando script...')
  const page = await browser.newPage()

  await page.goto('https://www.bet365.es/#/IP/B1');

  etc...

Versions

System: OS: Windows 10 10.0.18363 CPU: (8) x64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz Memory: 8.31 GB / 15.86 GB Binaries: Node: 12.17.0 - C:\Program Files\nodejs\node.EXE npm: 6.14.10 - ~\AppData\Roaming\npm\npm.CMD

[vladtreny]

example simulating the problem.

Try to purge or remove userDataDir .

[FabianoDevX]

example simulating the problem.

Try to purge or remove userDataDir . But cannot reproduce

it worked, thank you.

[vladtreny]

Seems like cached somewhere in cookies or local storage that you are a bot

[FabianoDevX]

Seems like cached somewhere in cookies or local storage that you are a bot

delete folder and tried again

[weizh0411]

@vladtreny i made the more test.

The packages what i use are:

"puppeteer-core": "^3.3.0", "puppeteer-extra": "^3.1.15", "puppeteer-extra-plugin-stealth": "^2.6.5"

1. On windows i don't have problem to open 365. (Chrome Version 88.0.4324.146)
2. i can't open it on linux. (Chrome Version 88.0.4324.150 and Version 81.0.4044.138)

The launch options are:

const launchOptions = Object.assign({
      dumpio: false, headless: false, defaultViewport: null, devtools: false,
      args: [
        `--disable-dev-shm-usage`, `--force-device-scale-factor=1`, `--disable-gpu`,
        `--disable-accelerated-2d-canvas`, `--lang=en-GB`, `--allow-insecure-localhost`,
      ]
        .concat(this.isWindows ? [
          `--start-maximized`, `--user-data-dir=D:\\AppData\\Puppeteer\\Master`,
        ] : [
          `--no-sandbox`, `--disable-setuid-sandbox`,
          `--window-position=0,0`, `--window-size=1366,768`, `--user-data-dir`,
        ]),
      ignoreDefaultArgs: [
        // `--enable-automation`, `--remote-debugging-port=0`,
      ]
    }, (this.isWindows ? {
      executablePath: "C:/Program Files (x86)/Google/Chrome/Application/chrome.exe",
    } : {
      executablePath: "/opt/google/chrome/chrome",
    }));

Because i open the browser first and maybe not open the 365 page with bot. I add the workaround on event listener.

this.browser.on('targetcreated', async (target) => {
      if (target.type() === 'page') {
        const page = await target.page();
        await page.evaluateOnNewDocument(() => {
          Object.defineProperty(navigator, 'maxTouchPoints', {
            get() {
                return 1;
            },
          });

          navigator.permissions.query = i => ({then: f => f({state: "prompt", onchange: null})});
        });
      } 
    });

"iframe.contentWindow" and "chrome.runtime" are removed from stealth.

[weizh0411]

test also with the current packages of puppeteer-core, puppeteer-extra and puppeteer-extra-plugin-stealth, it didn't work on linux.

[ldrapeau]

test also with the current packages of puppeteer-core, puppeteer-extra and puppeteer-extra-plugin-stealth, it didn't work on linux.

It works when I do it manually from my chrome browser in windows, can't we tell puppeteer to "join" the copy of my browser that is open and take control from their?

[filetopaixao]

@vladtreny can get X-Net-Sync-Term and make a request to SportsBook.API, but I would like the statistics of live matches, like goal kicks, dangerous attacks, corners, etc ... How do I get this?

[weizh0411]

test also with the current packages of puppeteer-core, puppeteer-extra and puppeteer-extra-plugin-stealth, it didn't work on linux.

It works when I do it manually from my chrome browser in windows, can't we tell puppeteer to "join" the copy of my browser that is open and take control from their?

There's no problem to open or control 365 in windows now, you don't need "join" and "controal".

My Problem is how to make it work on my linux.

ps: now it works on windows without navigator.maxTouchPoints also.

[ldrapeau]

test also with the current packages of puppeteer-core, puppeteer-extra and puppeteer-extra-plugin-stealth, it didn't work on linux.

It works when I do it manually from my chrome browser in windows, can't we tell puppeteer to "join" the copy of my browser that is open and take control from their?

There's no problem to open or control 365 in windows now, you don't need "join" and "controal".

My Problem is how to make it work on my linux.

ps: now it works on windows without navigator.maxTouchPoints also.

Hello,

I'm curious how "There's no problem to open or control 365 in windows now" is your case, because my program below only displays a white page that never loads on my windows 10 computer, and just a couple of days ago it worked. Thanks for all your input, Louis

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

const stealth = StealthPlugin();

stealth.enabledEvasions.delete('chrome.runtime');
stealth.enabledEvasions.delete('iframe.contentWindow');
puppeteer.use(stealth);

const second = 1.0;

async function nav() {

  const browser = await puppeteer.launch({headless: false, slowMo: 200} ); 
  const page = await browser.newPage();

  await page.evaluateOnNewDocument(() => {

    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

    navigator.permissions.query = i => ({then: f => f({state: "prompt", onchange: null})});

    });

  await page.goto('https://www.bet365.com');
  await page.waitFor(15000*second);

  await browser.close();
  }

nav();

[ldrapeau]

As a side note, I also removed "navigator.maxTouchPoints" and it produces the same result. These are my current versions:

C:\Users\louis\node>show_versions.bat
npm show chromium version
3.0.2
npm show puppeteer version
7.0.1
npm show puppeteer-extra version
3.1.16
npm show puppeteer-extra-plugin-stealth version
2.7.4

C:\Users\louis\node>

[weizh0411]

i found the reason, use yarn but not npm. it solve my problem.

[berstend]

I'm locking this issue now for the following reasons:

• We're maintaining a general purpose plugin framework for puppeteer (and soon playwright). One of the available plugins is stealth, with the stated purpose to remove subtle differences between scripted & regular browsers
• We don't do (or condone) site-specific modifications and will never have them in the plugin, our goal is to develop generic patches to remove differences between scripted and regular browsers
• Our issue tracker is meant for technical discussions among developers and not a support forum for end users
• If you find something in stealth that needs patching please do your homework and open a technical PR as per the template
• If you need to discuss usage of this plugin with others please use the community discord

Thanks!