[Bug] bet365.com stopped working 2 days ago

apulidoc commented 3 years ago

edit: Further discussion should happen on the community discord

Describe the bug

Im trying to scrap bet365 website
Expect open the website bet365.com without problems.
I was using puppeteer-extra-plugin-stealth for this purpose but is not working anymore from 2 days ago.

Code Snippet

const fs = require('fs')
const json = require('JSON')
const puppeteer = require('puppeteer-extra')
const { Bot } = require('tgapi')
const mysql = require('mysql')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')

var dbConfig ={
  host     : 'localhost',
  port     : '3307',
  user     : 'xxxx',
  password : 'xxxx',
  database : 'mydb'
}

let connection = mysql.createPool(dbConfig);
connection.setMaxListeners(20)

puppeteer.use(StealthPlugin())

//Ejecucion de Puppeteer
puppeteer.launch({ args: ['--start-maximized'], headless: false }).then(async browser => {
  console.log('Ejecutando script...')
  const page = await browser.newPage()

  await page.goto('https://www.bet365.es/#/IP/B1');

  etc...

Versions

System: OS: Windows 10 10.0.18363 CPU: (8) x64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz Memory: 8.31 GB / 15.86 GB Binaries: Node: 12.17.0 - C:\Program Files\nodejs\node.EXE npm: 6.14.10 - ~\AppData\Roaming\npm\npm.CMD

Screenshot_7

Skllipzgn commented 3 years ago

I observe the same thing. There is such an error in the console

hrehman200 commented 3 years ago

Same thing here ... can't load bet365.com

prescience-data commented 3 years ago

Which Chrome version are you using?

hrehman200 commented 3 years ago

Chromium the one which is installed alongside puppeteer.

apulidoc commented 3 years ago

Which Chrome version are you using?

Chromium Versión 88.0.4298.0

Also i downgraded to 86.0.4240.0 but still is not working

apulidoc commented 3 years ago

Removing stealth plugin and adding '--disable-blink-features=AutomationControlled' its working again

hrehman200 commented 3 years ago

Thanks @apulidoc, its working.

Niek commented 3 years ago

The bet365 site has very strict IP checks. If you open the site with an:

IP from a country that is not supported: you get this error - https://www.bet365.com/Members/helpers/error.aspx
IP from a datacenter range: they load a JS script that is doing a bunch of calls to localhost which are blocked by Chrome. I tested a non-puppeteer Chrome and Safari but I have the same issue there.
IP from a residential range: it hangs loading when using puppeteer, no matter if stealth is enabled or not.

Some more observations:

Using the bundled Chromium from puppeteer directly works fine:

./node_modules/puppeteer/.local-chromium/mac-818858/chrome-mac/Chromium.app/Contents/MacOS/Chromium https://www.bet365.com/ --no-first-run --user-data-dir=$(mktemp)

Connecting puppeteer to an existing running browser (started with --remote-debugging-port=9992) with puppeteer.connect({browserURL: 'http://localhost:9992'}): it's again stuck on loading the site. This means they detect something in the way the page is opened.
If you open an existing page (so not opened by puppeteer) in a browser opened with stealth (to avoid the webdriver property) it works fine - obviously page-level stealth evasions are not applied but it's working fine. Here's working sample code:
```
const puppeteer = require('puppeteer-extra')
puppeteer.use(require('puppeteer-extra-plugin-stealth')())
```

puppeteer.launch({ headless: false }).then(async browser => { const page = (await browser.pages())[0];

await page.goto('https://www.bet365.com/'); });


We need to investigate what they detect exactly in newly opened pages.

prescience-data commented 3 years ago

We need to investigate what they detect exactly in newly opened pages.

My immediate thoughts is it must be one of the evasions leaking / mismatching if it's occurring only on newPage() and passing on bundled Chromium?

(ie something is out of alignment with real-Chrome)?

Niek commented 3 years ago

I'm in favor of deleting those evasions, since they do more wrong than good. I think @berstend was already planning to remove them in automation-extra, but it might be a good idea to also remove them in the existing code.

berstend commented 3 years ago

Seems like the issue it was targeting was solved in recent releases of pptr or it was fixed in chrome itself.. If the TS says it broke 2 days ago

Do you have any link to that?

I think we might be confusing different issues here:

https://github.com/puppeteer/puppeteer/issues/1106
- This is the srcdoc iframe bug that's the reason for iframe.contentWindow to exist
https://github.com/puppeteer/puppeteer/issues/3667
- This is another longstanding bug that affects implicitly created targets
- Chromium Bug with some recent activity: https://bugs.chromium.org/p/chromium/issues/detail?id=1070568

berstend commented 3 years ago

Workaround for the time being:

const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

berstend commented 3 years ago

I confirmed through local testing that the bug with iframes is still present. There's potential to improve the iframe.contentWindow evasion though, so it doesn't break e.g. bet356

kingname commented 3 years ago

@berstend As your code is based on puppeteer, if I want to use Selenium, what is the equivalent of stealth.enabledEvasions.delete in Selenium?

berstend commented 3 years ago

@berstend As your code is based on puppeteer, if I want to use Selenium, what is the equivalent of stealth.enabledEvasions.delete in Selenium?

Selenium is not supported by us - stealth.enabledEvasions is a Map() which is used by the stealth plugin (.delete is just a native method of Maps). This won't help you as the puppeteer-extra as well as the stealth plugin are not written to support selenium. :-)

berstend commented 3 years ago

Closing for now, as iframe.contentWindow issues are quite well known by now and a workaround is mentioned above. Eventually we improve the evasion so it doesn't cause as much breakage.

kasd7 commented 3 years ago

It seems like Bet365 has improved its detection method, and now the workaround doesn't work.

Example of use:

const puppeteer = require('puppeteer-extra')

const StealthPlugin = require('puppeteer-extra-plugin-stealth')

const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

puppeteer.launch({ args: ['--no-sandbox','--start-maximized'], headless: false }).then(async browser => {
  const page = await browser.newPage()

  await page.goto('https://www.bet365.com/');
  await page.waitFor(50_000)
})

@berstend maybe we should reopen the issue?

migue74 commented 3 years ago

Yup, bet365 broke puppeteer again :(

helderppb commented 3 years ago

Yes! It has stopped just today for me. Does anyone has a new workaround to solve it?

apulidoc commented 3 years ago

if anyone find a solution please post here! =)

vladtreny commented 3 years ago

Hello friends,

Here I've created a temporary fix for bet365

Run this and it should work

        await page.evaluateOnNewDocument(() => {
            Object.defineProperty(navigator, 'maxTouchPoints', {
                get() {
                    return 1;
                },
            });

        });

Will take a closer look on free time

helderppb commented 3 years ago

Do you have the complete code that you're using? I'm using the code below and it's not working.

const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => { console.log('Running tests..'); const page = await browser.newPage();

await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');

});

vladtreny commented 3 years ago

Do you have the complete code that you're using? I'm using the code below and it's not working.

const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => { console.log('Running tests..'); const page = await browser.newPage();
await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');
});

Do not forget the rest stuff

const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);

helderppb commented 3 years ago

It’s not working for me. Can you share the code you’re using, please?

const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); const stealth = StealthPlugin(); stealth.enabledEvasions.delete('chrome.runtime') stealth.enabledEvasions.delete('iframe.contentWindow') puppeteer.use(stealth);

puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {

console.log('Running tests..');
const page = await browser.newPage();

await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');

});

vladtreny commented 3 years ago

It’s not working for me. Can you share the code you’re using, please?

const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); const stealth = StealthPlugin(); stealth.enabledEvasions.delete('chrome.runtime') stealth.enabledEvasions.delete('iframe.contentWindow') puppeteer.use(stealth);

puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {
console.log('Running tests..');
const page = await browser.newPage();

await page.goto('https://bet365.com');

await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            return 1;
        },
    });

});

await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');
});

You need to run await page.evaluateOnNewDocument(() => { before await page.goto('https://bet365.com');

apulidoc commented 3 years ago

Im having troubles with this workaround...

When I try to make an a evaluate with queryselector or queryselectorall the pages freeze automatically and chromium begin to leak memory as fuck. I cant use the console webbrowser, i cant retrieve any data from the website...

Anyone has this trouble?

vladtreny commented 3 years ago

Friends, you can also use this one

            let n = 1;
            Object.defineProperty(navigator, 'maxTouchPoints', {
                get() {
                    setTimeout(() => n = 0, 0);
                    return n;
                },
            });

vladtreny commented 3 years ago

Im having troubles with this workaround...

When I try to make an a evaluate with queryselector or queryselectorall the pages freeze automatically and chromium begin to leak memory as fuck. I cant use the console webbrowser, i cant retrieve any data from the website...

Anyone has this trouble?

Try on await page.evaluateOnNewDocument(() => { add windows.qs = document.querySelector

then, on every evaluate

.evaluate(()=>{
document.querySelector = windows.qs
document.querySelector('...')

apulidoc commented 3 years ago

Im having troubles with this workaround... When I try to make an a evaluate with queryselector or queryselectorall the pages freeze automatically and chromium begin to leak memory as fuck. I cant use the console webbrowser, i cant retrieve any data from the website... Anyone has this trouble?

Try on await page.evaluateOnNewDocument(() => { add windows.qs = document.querySelector

then, on every evaluate
.evaluate(()=>{
document.querySelector = windows.qs
document.querySelector('...')

Worked perfectly overriding again document.queryselector

gregjefferson commented 3 years ago

I'm always being banned by b365 30min-90min after first request, how do you guys avoid it? Proxies, what proxies eventually? Could you maybe DM me with your solution? Thanks

filetopaixao commented 3 years ago

Good night. I also cannot access the bet365 website after using the puppeteer. Is this temporary or permanent? How do I get the requests that the bet365 page makes and its headers?

I thank you for your help. =)

weizh0411 commented 3 years ago

Can't open bet365 again, has anyone the same problem?

hrehman200 commented 3 years ago

Yes it suddenly stopped working. I can open a live game, but after a couple of seconds it gets stuck.

On Sat, Feb 6, 2021 at 12:08 AM weizh0411 notifications@github.com wrote:

Can't open bet365 again, has anyone the same problem?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/berstend/puppeteer-extra/issues/399#issuecomment-774230147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUWZ5EK25CAW4WCWQVXKALS5Q6Z3ANCNFSM4VCSSF2Q .

-- Regards Haris-ur-Rehman

berstend commented 3 years ago

@weizh0411 @hrehman200 did you follow the workarounds mentioned in this thread (incl. caching document.querySelector)?

weizh0411 commented 3 years ago

@berstend yes

FabianoDevX commented 3 years ago

bet365 broke the puppeteer again, the site won't open, if you disable the stealth plugin I can open it, but the site keeps crashing

berstend commented 3 years ago

Yeah, as they seem to be actively monitoring our development there won't be changes made to the plugin right now. In case you depend on that site working with stealth I recommend asking nicely in our discord channels, a few community members were actively working on solutions (and succeeded from what I can tell).

berstend commented 3 years ago

@FabianoDevX either you didn't try the querySelector workaround, you didn't read the other comments here or you didn't mention trying it - none of these are good netiquette. ;-)

FabianoDevX commented 3 years ago

@berstend Yes,I tried all the alternatives on this topic, until today in the morning it was working perfectly.

berstend commented 3 years ago

@berstend Yes,I tried all the alternatives on this topic, until today in the morning it was working perfectly.

None of this you mentioned in your first comment, I recommend adding as much context as possible if you depend on others helping you.

FabianoDevX commented 3 years ago

@berstend

const puppeteer = require("puppeteer-extra");
const path = require("path");

const StealthPlugin = require("puppeteer-extra-plugin-stealth")();
StealthPlugin.enabledEvasions.delete("chrome.runtime");
StealthPlugin.enabledEvasions.delete("iframe.contentWindow");
puppeteer.use(StealthPlugin);

(async () => {

const pupeteerBrowser = await puppeteer.launch({
  headless: false,
  userDataDir: path.resolve(__dirname, "./perfil"),
  args: [
    "--disable-infobars",
    "--no-sandbox",
    "--disable-blink-features=AutomationControlled",
  ],
  ignoreDefaultArgs: ["--enable-automation"],
});

// this.pupeteerBrowser = await puppeteer.connect({browserURL: 'http://localhost:9992'})

/**@type  import('puppeteer').Page **/
const pupeteerPage = await pupeteerBrowser.newPage();

await pupeteerPage.evaluateOnNewDocument(() => {
  let n = 1;

  Object.defineProperty(navigator, "maxTouchPoints", {
    get() {
      setTimeout(() => (n = 0), 0);
      return n;
    },
  });
});

await pupeteerPage.evaluateOnNewDocument(() => {
  window.qs = document.querySelector;
  window.qsAll = document.querySelectorAll;
});

await pupeteerPage.goto("https://bet365.com/", { waitUntil: "networkidle0" });
//await this.pupeteerPage.setViewport({ width: 1200, height: 720 });

await pupeteerPage.evaluate(() => {
  document.querySelector = window.qs;
  document.querySelectorAll = window.qsAll;
});

})();

the site is forever loading with a blank page error

vladtreny commented 3 years ago

Hello friends, I've created a new fix.

await page.evaluateOnNewDocument(() => {

    Object.defineProperty(navigator, 'maxTouchPoints', {
        get() {
            "¯\_(ツ)_/¯";
            return 1;
        },
    });

    "✌(-‿-)✌";
    navigator.permissions.query = i => ({then: f => f({state: "prompt", onchange: null})});

});

For Python users, put it to like "Page.addScriptToEvaluateOnNewDocument"

weizh0411 commented 3 years ago

@vladtreny it works, thanks for the fix.

weizh0411 commented 3 years ago

365 kill it again.

kingname commented 3 years ago

The programmer of bet365 is monitoring this issue. Everyone please do not update solution here.

kingname commented 3 years ago

Sure. So we have 2 way to go.

Find a method which will not be blocked even if they know what we have done.
Talk about the solution in a safe place.

vladtreny commented 3 years ago

365 kill it again.

Are you sure? I cannot reproduce it

FabianoDevX commented 3 years ago

when I try to change location.hash , the system crashes, does not leave this screen.

vladtreny commented 3 years ago

when I try to change location.hash, the system crashes, does not leave this screen.

Hello, Provide code example to reproduce

FabianoDevX commented 3 years ago

@vladtreny example simulating the problem.


const puppeteer = require("puppeteer-extra");
const path = require("path");

const StealthPlugin = require("puppeteer-extra-plugin-stealth")();
StealthPlugin.enabledEvasions.delete("chrome.runtime");
StealthPlugin.enabledEvasions.delete("iframe.contentWindow");
puppeteer.use(StealthPlugin);

(async () => {

const puppeteerBrowser = await puppeteer.launch({
  headless: false,
  userDataDir: path.resolve(__dirname, "./perfil"),
  args: [
    "--disable-infobars",
    "--no-sandbox",
    "--disable-blink-features=AutomationControlled",
  ],
  ignoreDefaultArgs: ["--enable-automation"],
});

// this.pupeteerBrowser = await puppeteer.connect({browserURL: 'http://localhost:9992'})

/**@type  import('puppeteer').Page **/
const puppeteerPage = await puppeteerBrowser.newPage();

await puppeteerPage.evaluateOnNewDocument(() => {
  window.qs = document.querySelector;
  window.qsAll = document.querySelectorAll;

  let n = 1;

  Object.defineProperty(navigator, "maxTouchPoints", {
    get() {
      setTimeout(() => (n = 0), 0);
      return n;
    },
  });

  navigator.permissions.query = i => ({then: f => f({state: "prompt", onchange: null})});
});

await puppeteerPage.goto("https://bet365.com/", { waitUntil: "networkidle0" });
//await this.pupeteerPage.setViewport({ width: 1200, height: 720 });

await puppeteerPage.evaluate(() => {
  document.querySelector = window.qs;
  document.querySelectorAll = window.qsAll;
});

await puppeteerPage.evaluate(() => {
  location.hash = '/AS/B1/';
});

await puppeteerPage.waitForTimeout(3000);

await puppeteerPage.evaluate(() => {
  location.hash = '/AC/B1/C1/D13/E108/F16/';
});

})();

berstend / puppeteer-extra

[Bug] bet365.com stopped working 2 days ago #399

edit: Further discussion should happen on the community discord