Closed apulidoc closed 3 years ago
I observe the same thing. There is such an error in the console
Same thing here ... can't load bet365.com
Which Chrome version are you using?
Chromium the one which is installed alongside puppeteer.
Which Chrome version are you using?
Chromium Versión 88.0.4298.0
Also i downgraded to 86.0.4240.0 but still is not working
Removing stealth plugin and adding '--disable-blink-features=AutomationControlled' its working again
Thanks @apulidoc, its working.
The bet365 site has very strict IP checks. If you open the site with an:
Some more observations:
./node_modules/puppeteer/.local-chromium/mac-818858/chrome-mac/Chromium.app/Contents/MacOS/Chromium https://www.bet365.com/ --no-first-run --user-data-dir=$(mktemp)
--remote-debugging-port=9992
) with puppeteer.connect({browserURL: 'http://localhost:9992'})
: it's again stuck on loading the site. This means they detect something in the way the page is opened.
const puppeteer = require('puppeteer-extra')
puppeteer.use(require('puppeteer-extra-plugin-stealth')())
puppeteer.launch({ headless: false }).then(async browser => { const page = (await browser.pages())[0];
await page.goto('https://www.bet365.com/'); });
We need to investigate what they detect exactly in newly opened pages.
We need to investigate what they detect exactly in newly opened pages.
My immediate thoughts is it must be one of the evasions leaking / mismatching if it's occurring only on newPage()
and passing on bundled Chromium?
(ie something is out of alignment with real-Chrome)?
I'm in favor of deleting those evasions, since they do more wrong than good. I think @berstend was already planning to remove them in automation-extra
, but it might be a good idea to also remove them in the existing code.
Seems like the issue it was targeting was solved in recent releases of pptr or it was fixed in chrome itself.. If the TS says it broke 2 days ago
Do you have any link to that?
I think we might be confusing different issues here:
https://github.com/puppeteer/puppeteer/issues/1106
iframe.contentWindow
to existhttps://github.com/puppeteer/puppeteer/issues/3667
Workaround for the time being:
const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);
I confirmed through local testing that the bug with iframes is still present. There's potential to improve the iframe.contentWindow
evasion though, so it doesn't break e.g. bet356
@berstend As your code is based on puppeteer, if I want to use Selenium, what is the equivalent of stealth.enabledEvasions.delete
in Selenium?
@berstend As your code is based on puppeteer, if I want to use Selenium, what is the equivalent of
stealth.enabledEvasions.delete
in Selenium?
Selenium is not supported by us - stealth.enabledEvasions
is a Map()
which is used by the stealth plugin (.delete
is just a native method of Maps). This won't help you as the puppeteer-extra
as well as the stealth plugin are not written to support selenium. :-)
Closing for now, as iframe.contentWindow
issues are quite well known by now and a workaround is mentioned above. Eventually we improve the evasion so it doesn't cause as much breakage.
It seems like Bet365 has improved its detection method, and now the workaround doesn't work.
Example of use:
const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);
puppeteer.launch({ args: ['--no-sandbox','--start-maximized'], headless: false }).then(async browser => {
const page = await browser.newPage()
await page.goto('https://www.bet365.com/');
await page.waitFor(50_000)
})
@berstend maybe we should reopen the issue?
Yup, bet365 broke puppeteer again :(
Yes! It has stopped just today for me. Does anyone has a new workaround to solve it?
if anyone find a solution please post here! =)
Hello friends,
Here I've created a temporary fix for bet365
Run this and it should work
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'maxTouchPoints', {
get() {
return 1;
},
});
});
Will take a closer look on free time
Do you have the complete code that you're using? I'm using the code below and it's not working.
const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => { console.log('Running tests..'); const page = await browser.newPage();
await page.goto('https://bet365.com');
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'maxTouchPoints', {
get() {
return 1;
},
});
});
await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');
});
Do you have the complete code that you're using? I'm using the code below and it's not working.
const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => { console.log('Running tests..'); const page = await browser.newPage();
await page.goto('https://bet365.com'); await page.evaluateOnNewDocument(() => { Object.defineProperty(navigator, 'maxTouchPoints', { get() { return 1; }, }); }); await page.screenshot({ path: 'bet365.png', fullPage: true }); await browser.close(); console.log('All done, check the screenshot.');
});
Do not forget the rest stuff
const stealth = StealthPlugin();
stealth.enabledEvasions.delete('chrome.runtime')
stealth.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(stealth);
It’s not working for me. Can you share the code you’re using, please?
const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); const stealth = StealthPlugin(); stealth.enabledEvasions.delete('chrome.runtime') stealth.enabledEvasions.delete('iframe.contentWindow') puppeteer.use(stealth);
puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {
console.log('Running tests..');
const page = await browser.newPage();
await page.goto('https://bet365.com');
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'maxTouchPoints', {
get() {
return 1;
},
});
});
await page.screenshot({ path: 'bet365.png', fullPage: true });
await browser.close();
console.log('All done, check the screenshot.');
});
It’s not working for me. Can you share the code you’re using, please?
const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); const stealth = StealthPlugin(); stealth.enabledEvasions.delete('chrome.runtime') stealth.enabledEvasions.delete('iframe.contentWindow') puppeteer.use(stealth);
puppeteer.launch({ executablePath: '/usr/bin/chromium-browser', headless: true, args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'] }).then(async browser => {
console.log('Running tests..'); const page = await browser.newPage(); await page.goto('https://bet365.com'); await page.evaluateOnNewDocument(() => { Object.defineProperty(navigator, 'maxTouchPoints', { get() { return 1; }, }); }); await page.screenshot({ path: 'bet365.png', fullPage: true }); await browser.close(); console.log('All done, check the screenshot.');
});
You need to run await page.evaluateOnNewDocument(() => {
before await page.goto('https://bet365.com');
Im having troubles with this workaround...
When I try to make an a evaluate with queryselector or queryselectorall the pages freeze automatically and chromium begin to leak memory as fuck. I cant use the console webbrowser, i cant retrieve any data from the website...
Anyone has this trouble?
Friends, you can also use this one
let n = 1;
Object.defineProperty(navigator, 'maxTouchPoints', {
get() {
setTimeout(() => n = 0, 0);
return n;
},
});
Im having troubles with this workaround...
When I try to make an a evaluate with queryselector or queryselectorall the pages freeze automatically and chromium begin to leak memory as fuck. I cant use the console webbrowser, i cant retrieve any data from the website...
Anyone has this trouble?
Try on await page.evaluateOnNewDocument(() => {
add windows.qs = document.querySelector
then, on every evaluate
.evaluate(()=>{
document.querySelector = windows.qs
document.querySelector('...')
Im having troubles with this workaround... When I try to make an a evaluate with queryselector or queryselectorall the pages freeze automatically and chromium begin to leak memory as fuck. I cant use the console webbrowser, i cant retrieve any data from the website... Anyone has this trouble?
Try on
await page.evaluateOnNewDocument(() => {
addwindows.qs = document.querySelector
then, on every
evaluate
.evaluate(()=>{ document.querySelector = windows.qs document.querySelector('...')
Worked perfectly overriding again document.queryselector
I'm always being banned by b365 30min-90min after first request, how do you guys avoid it? Proxies, what proxies eventually? Could you maybe DM me with your solution? Thanks
Good night. I also cannot access the bet365 website after using the puppeteer. Is this temporary or permanent? How do I get the requests that the bet365 page makes and its headers?
I thank you for your help. =)
Can't open bet365 again, has anyone the same problem?
Yes it suddenly stopped working. I can open a live game, but after a couple of seconds it gets stuck.
On Sat, Feb 6, 2021 at 12:08 AM weizh0411 notifications@github.com wrote:
Can't open bet365 again, has anyone the same problem?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/berstend/puppeteer-extra/issues/399#issuecomment-774230147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUWZ5EK25CAW4WCWQVXKALS5Q6Z3ANCNFSM4VCSSF2Q .
-- Regards Haris-ur-Rehman
@weizh0411 @hrehman200 did you follow the workarounds mentioned in this thread (incl. caching document.querySelector
)?
@berstend yes
bet365 broke the puppeteer again, the site won't open, if you disable the stealth plugin I can open it, but the site keeps crashing
Yeah, as they seem to be actively monitoring our development there won't be changes made to the plugin right now. In case you depend on that site working with stealth I recommend asking nicely in our discord channels, a few community members were actively working on solutions (and succeeded from what I can tell).
@FabianoDevX either you didn't try the querySelector
workaround, you didn't read the other comments here or you didn't mention trying it - none of these are good netiquette. ;-)
@berstend Yes,I tried all the alternatives on this topic, until today in the morning it was working perfectly.
@berstend Yes,I tried all the alternatives on this topic, until today in the morning it was working perfectly.
None of this you mentioned in your first comment, I recommend adding as much context as possible if you depend on others helping you.
@berstend
const puppeteer = require("puppeteer-extra");
const path = require("path");
const StealthPlugin = require("puppeteer-extra-plugin-stealth")();
StealthPlugin.enabledEvasions.delete("chrome.runtime");
StealthPlugin.enabledEvasions.delete("iframe.contentWindow");
puppeteer.use(StealthPlugin);
(async () => {
const pupeteerBrowser = await puppeteer.launch({
headless: false,
userDataDir: path.resolve(__dirname, "./perfil"),
args: [
"--disable-infobars",
"--no-sandbox",
"--disable-blink-features=AutomationControlled",
],
ignoreDefaultArgs: ["--enable-automation"],
});
// this.pupeteerBrowser = await puppeteer.connect({browserURL: 'http://localhost:9992'})
/**@type import('puppeteer').Page **/
const pupeteerPage = await pupeteerBrowser.newPage();
await pupeteerPage.evaluateOnNewDocument(() => {
let n = 1;
Object.defineProperty(navigator, "maxTouchPoints", {
get() {
setTimeout(() => (n = 0), 0);
return n;
},
});
});
await pupeteerPage.evaluateOnNewDocument(() => {
window.qs = document.querySelector;
window.qsAll = document.querySelectorAll;
});
await pupeteerPage.goto("https://bet365.com/", { waitUntil: "networkidle0" });
//await this.pupeteerPage.setViewport({ width: 1200, height: 720 });
await pupeteerPage.evaluate(() => {
document.querySelector = window.qs;
document.querySelectorAll = window.qsAll;
});
})();
the site is forever loading with a blank page
Hello friends, I've created a new fix.
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'maxTouchPoints', {
get() {
"¯\_(ツ)_/¯";
return 1;
},
});
"✌(-‿-)✌";
navigator.permissions.query = i => ({then: f => f({state: "prompt", onchange: null})});
});
For Python users, put it to like "Page.addScriptToEvaluateOnNewDocument"
@vladtreny it works, thanks for the fix.
365 kill it again.
The programmer of bet365 is monitoring this issue. Everyone please do not update solution here.
Sure. So we have 2 way to go.
365 kill it again.
Are you sure? I cannot reproduce it
when I try to change location.hash
, the system crashes, does not leave this screen.
when I try to change
location.hash
, the system crashes, does not leave this screen.
Hello, Provide code example to reproduce
@vladtreny example simulating the problem.
const puppeteer = require("puppeteer-extra");
const path = require("path");
const StealthPlugin = require("puppeteer-extra-plugin-stealth")();
StealthPlugin.enabledEvasions.delete("chrome.runtime");
StealthPlugin.enabledEvasions.delete("iframe.contentWindow");
puppeteer.use(StealthPlugin);
(async () => {
const puppeteerBrowser = await puppeteer.launch({
headless: false,
userDataDir: path.resolve(__dirname, "./perfil"),
args: [
"--disable-infobars",
"--no-sandbox",
"--disable-blink-features=AutomationControlled",
],
ignoreDefaultArgs: ["--enable-automation"],
});
// this.pupeteerBrowser = await puppeteer.connect({browserURL: 'http://localhost:9992'})
/**@type import('puppeteer').Page **/
const puppeteerPage = await puppeteerBrowser.newPage();
await puppeteerPage.evaluateOnNewDocument(() => {
window.qs = document.querySelector;
window.qsAll = document.querySelectorAll;
let n = 1;
Object.defineProperty(navigator, "maxTouchPoints", {
get() {
setTimeout(() => (n = 0), 0);
return n;
},
});
navigator.permissions.query = i => ({then: f => f({state: "prompt", onchange: null})});
});
await puppeteerPage.goto("https://bet365.com/", { waitUntil: "networkidle0" });
//await this.pupeteerPage.setViewport({ width: 1200, height: 720 });
await puppeteerPage.evaluate(() => {
document.querySelector = window.qs;
document.querySelectorAll = window.qsAll;
});
await puppeteerPage.evaluate(() => {
location.hash = '/AS/B1/';
});
await puppeteerPage.waitForTimeout(3000);
await puppeteerPage.evaluate(() => {
location.hash = '/AC/B1/C1/D13/E108/F16/';
});
})();
edit: Further discussion should happen on the community discord
Describe the bug
Code Snippet
Versions
System: OS: Windows 10 10.0.18363 CPU: (8) x64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz Memory: 8.31 GB / 15.86 GB Binaries: Node: 12.17.0 - C:\Program Files\nodejs\node.EXE npm: 6.14.10 - ~\AppData\Roaming\npm\npm.CMD