Closed entrptaher closed 6 years ago
So the issue is that headless is triggering the capchta page? What happens if you launch with full chrome?
On Sun, Oct 1, 2017, 9:39 AM Abu Taher notifications@github.com wrote:
Steps to reproduce
Tell us about your environment:
- Puppeteer version: v0.11.0
- Platform / OS version:
- URLs (if applicable): https://www.upwork.com/ab/account-security/login
What steps will reproduce the problem?
Create a script that will go to upwork, and login. Maybe first time it'll load properly, it'll fail to load on second try.
Please include code that reproduces the issue.
- Works fine on Chromeless
const chromeless = new Chromeless({ remote: true })const screenshot = await chromeless .goto('https://www.upwork.com/ab/account-security/login') .type("", "#login_username") .type("","#login_password") .click("#layout > div.container > div > form > div.form-group.hidden-xs.m-0-bottom > button") .wait(20000) .screenshot()console.log(screenshot)await chromeless.end()
- Does not work at in puppeteer
const browser = await puppeteer.launch();const page = await browser.newPage(); try{await page.goto('https://www.upwork.com/ab/account-security/login');await page.click("#login_username")await page.type("")await page.click("#login_password")await page.type("")await page.click("#layout > div.container > div > form > div.form-group.hidden-xs.m-0-bottom > button")await page.waitFor(2000) await page.screenshot({path: 'screenshot.png'});
}catch(e){ console.log(e); await page.screenshot({path: 'screenshot.png'}); }await browser.close();
What is the expected result? Disclaimer: The account used was a dummy account.
The page will be logged in nicely. [image: image] https://user-images.githubusercontent.com/8284972/31058199-9ea9ab7e-a711-11e7-95a4-84c2bb6b0b9a.png [image: image] https://user-images.githubusercontent.com/8284972/31058229-0a90f8b0-a712-11e7-9be3-0927b278153b.png
What happens instead? The page does not load properly, let alone login. [image: image] https://user-images.githubusercontent.com/8284972/31058201-a408c92e-a711-11e7-9a16-9829a25ecf97.png
[image: image] https://user-images.githubusercontent.com/8284972/31058228-008982ba-a712-11e7-890b-d41d5769433e.png
Additional Info:
-
Did you try it in local? Yes. Same result.
Did you try to use proxies? Yes. Same result.
Does it happen if you go to that page normally with your browser? No. The page loads normally.
Why compare to chromeless? That is using chromium and passing the problem pretty easy, while puppeteer is not. Notice how recaptcha is screaming about upgrading browser on puppeteer.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/puppeteer/issues/927, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOigFvbbV7B_7Yjx0D1oyOw6SWBF-Ocks5sn-p1gaJpZM4PqARB .
If I used my own chrome/firefox, no problem happens.
That's what I suspected. I've seen other sites think headless Chrome (and using it in browser automation) as a bot. We've seen that over in Lighthouse too.
Captchas tend to trigger on all sorts of things (suspect mouse movement, suspect browsing history, etc.) Unfortunately, they don't give us much more than that for good reason :p
If I normally use puppeteer {headless:false}
,
I made sure to use different proxies, without proxies, with proxies, etc etc.
Received signal 11 SEGV_MAPERR 000000000020
#0 0x564d00d2c337 base::debug::StackTrace::StackTrace()
#1 0x564d00d2beaf base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7fa49d605390 <unknown>
#3 0x564cfffc2cbe device::PlatformSensorProviderBase::MapSharedBufferForType()
#4 0x564cfffc7803 device::PlatformSensorProviderLinux::CreateSensorAndNotify()
#5 0x564cfffc77aa device::PlatformSensorProviderLinux::ProcessStoredRequests()
#6 0x564d00d2ca6b base::debug::TaskAnnotator::RunTask()
#7 0x564d00d447fb base::MessageLoop::RunTask()
#8 0x564d00d44e6b base::MessageLoop::DoWork()
#9 0x564d00d46b79 base::MessagePumpLibevent::Run()
#10 0x564d00d63210 base::RunLoop::Run()
#11 0x564cffbc5996 content::BrowserThreadImpl::IOThreadRun()
#12 0x564cffbc5b91 content::BrowserThreadImpl::Run()
#13 0x564d00d87d09 base::Thread::ThreadMain()
#14 0x564d00d82a83 base::(anonymous namespace)::ThreadFunc()
#15 0x7fa49d5fb6ba start_thread
#16 0x7fa49735d3dd clone
r8: 0000000000000000 r9: 0000000000000000 r10: 0000000000000000 r11: 000000000000063c
r12: 0000000000000c10 r13: 00007fa480dca6b8 r14: 00007fa480dca398 r15: 0000000000000005
di: 00007fa480dca1a8 si: 0000000000000001 bp: 0000000000000000 bx: 000001fb3ff74398
dx: 00000000000001c0 ax: 0000000000000000 cx: 0000000000000000 sp: 00007fa480dca370
ip: 0000564cfffc2cbe efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000006
trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000020
[end of stack trace]
Calling _exit(1). Core file will not be generated.
And I searched for the crash, I found it's a chromium bug introduced after 59th version. Still, I wonder, why specific sites always makes the problem? and why only chromeless does not have the problem? and why only nightmare and puppeteer triggers the bot/crash etc.
Can you post a link to the crbug?
I followed along the versions, tried version 58, which does not have headless. Obviously it did not work. I was having problem with version 59-62 and I saw version 63. The crash is no more if I use the latest Version 63.0.3236.0 (Developer Build). Nor do I see the captcha. Also I am using the following arguments, the disable extensions part was suggested on various replies but it did not work on the mentioned versions above.
'--enable-logging --v=1', '--ignore-certificate-errors', '--disable-extensions'
Bug report and other related Links for research purpose in case someone/me has this problem in future:
And, more debugging gave me this right before the crash,
Sat, 07 Oct 2017 09:02:44 GMT puppeteer:session ◀ RECV {"method":"Network.requestWillBeSent","params":{"requestId":"14756.35","loaderId":"14756.2","documentURL":"https://www.upwork.com/ab/account-security/login","request":{"url":"https://collector-pxss13u803.perimeterx.net/api/v1/collector","method":"POST","headers":{"Referer":"https://www.upwork.com/ab/account-security/login","Origin":"https://www.upwork.com","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3198.0 Safari/537.36","Content-type":"application/x-www-form-urlencoded"},"postData":"payload=W3sidCI6IlBYMiIsImQiOnsiUFg2MyI6IkxpbnV4IHg4Nl82NCIsIlBYOTYiOiJodHRwczovL3d3dy51cHdvcmsuY29tL2FiL2FjY291bnQtc2VjdXJpdHkvbG9naW4ifX1d&appId=PXSs13U803&tag=v2.48&uuid=47de4300-ab3e-11e7-b5b9-7728bb3e0aff","mixedContentType":"none","initialPriority":"High","referrerPolicy":"no-referrer-when-downgrade"},"timestamp":25559.40582,"wallTime":1507366964.03026,"initiator":{"type":"script","stack":{"callFrames":[{"functionName":"nrWrapper","scriptId":"31","url":"https://www.upwork.com/ab/account-security/login","lineNumber":6,"columnNumber":11614},{"functionName":"_postXhr","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":14858},{"functionName":"sendActivities","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":14247},{"functionName":"sendClientActivities","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":10212},{"functionName":"bootstrap","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":9907},{"functionName":"Vn","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":10771},{"functionName":"","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":23785},{"functionName":"","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":23790}]}},"type":"XHR","frameId":"14756.1"}}
Received signal 11 SEGV_MAPERR 000000000020
#0 0x564911925337 base::debug::StackTrace::StackTrace()
#1 0x564911924eaf base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7f91a8c3a390 <unknown>
#3 0x564910bbbcbe device::PlatformSensorProviderBase::MapSharedBufferForType()
#4 0x564910bc0803 device::PlatformSensorProviderLinux::CreateSensorAndNotify()
#5 0x564910bc07aa device::PlatformSensorProviderLinux::ProcessStoredRequests()
#6 0x564911925a6b base::debug::TaskAnnotator::RunTask()
#7 0x56491193d7fb base::MessageLoop::RunTask()
#8 0x56491193de6b base::MessageLoop::DoWork()
#9 0x56491193fb79 base::MessagePumpLibevent::Run()
#10 0x56491195c210 base::RunLoop::Run()
#11 0x5649107be996 content::BrowserThreadImpl::IOThreadRun()
#12 0x5649107beb91 content::BrowserThreadImpl::Run()
#13 0x564911980d09 base::Thread::ThreadMain()
#14 0x56491197ba83 base::(anonymous namespace)::ThreadFunc()
#15 0x7f91a8c306ba start_thread
#16 0x7f91a29923dd clone
r8: 0000000000000000 r9: 0000000000000000 r10: 0000000000000000 r11: 000000000000063c
r12: 0000000000000c10 r13: 00007f918c3ff6b8 r14: 00007f918c3ff398 r15: 0000000000000005
di: 00007f918c3ff1a8 si: 0000000000000001 bp: 0000000000000000 bx: 00003e1ecf3511e8
dx: 00000000000001c0 ax: 0000000000000000 cx: 0000000000000000 sp: 00007f918c3ff370
ip: 0000564910bbbcbe efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000006
trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000020
[end of stack trace]
The script it's trying to load is https://client.perimeterx.net/PXSs13U803/main.min.js, which has bunch of PDF and Canvas related stuff.
If I did this,
await page.setRequestInterceptionEnabled(true);
page.on('request', request => {
if (request.url.indexOf('perimeterx') > -1) {
console.log(`request blocked`);
request.abort();
} else {
request.continue();
}
});
then, the page does not crash. But it's not like I can monitor for one url out of thousands all the time. and it's not like I can mention which chrome version to download when I install puppeteer...?
Maybe some of these triggers the crash?
var n = {};
n.PX31 = e,
n.PX32 = t;
var r = s(),
o = eo(),
a = Kr();
n.PX274 = a.canvasData,
n.PX275 = a.canvasWinding,
n.PX276 = o.canvasfp,
n.PX210 = o.webglRenderer,
n.PX209 = o.webglVendor,
n.PX277 = o.webGLVersion,
n.PX278 = o.shadingLangulageVersion,
n.PX279 = o.unmaskedVendor,
n.PX280 = o.unmaskedRenderer,
n.PX281 = o.extensions,
n.PX282 = o.webglParameters,
n.PX33 = s() - r,
n.PX168 = De(),
n.PX248 = We(window.document),
n.PX249 = We(window),
n.PX57 = i(),
n.PX264 = He(),
n.PX265 = je(),
n.PX266 = Ve(window),
n.PX89 = Ue(),
ue(n, "PX286", function() {
return window.devicePixelRatio || ""
}, ""),
ue(n, "PX287", function() {
return navigator.hardwareConcurrency
}, -1),
ue(n, "PX288", function() {
return !!window.localStorage
}, !1),
ue(n, "PX289", function() {
return !!window.indexedDB
}, !1),
ue(n, "PX290", function() {
return !!window.openDatabase
}, !1),
ue(n, "PX291", function() {
return !!document.body.addBehavior
}, !1),
ue(n, "PX292", function() {
return navigator.cpuClass
}),
ue(n, "PX293", function() {
return !!window.sessionStorage
}, !1);
I found similar issue on other login pages too. For example I tried to login to github using puppeteer and got same stack error. The normal browser works fine, so I believe something is missing here.
This is a related bug with similar stack trace: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1702501
And same with upwork, it fails after this line,
Sat, 21 Oct 2017 17:28:10 GMT puppeteer:session ◀ RECV {"method":"Network.requestWillBeSent","params":{"requestId":"4549.37","loaderId":"4549.4","documentURL":"https://github.com/","request":{"url":"https://collector.githubapp.com/collect","method":"POST","headers":{"Cache-Control":"max-age=0","Origin":"https://github.com","Referer":"https://github.com/","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.79 Chrome/61.0.3163.79 Safari/537.36","Content-Type":"application/x-www-form-urlencoded"},"postData":"","mixedContentType":"none","initialPriority":"VeryLow","referrerPolicy":"origin-when-cross-origin"},"timestamp":101881.974186,"wallTime":1508606890.56355,"initiator":{"type":"script","stack":{"callFrames":[{"functionName":"a","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":1,"columnNumber":31058},{"functionName":"","scriptId":"38","url":"https://assets-cdn.github.com/assets/github-b27b8e3cfb60b7141910086e009b0b884cfa3eae7920f5346168c67d274f357e.js","lineNumber":1,"columnNumber":24857},{"functionName":"ie.D","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":13033},{"functionName":"In.send","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":28173},{"functionName":"n.(anonymous function)","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":6607},{"functionName":"rr.v","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":145},{"functionName":"rr.D","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":31648},{"functionName":"or","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":200},{"functionName":"u","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":5484},{"functionName":"","scriptId":"38","url":"https://assets-cdn.github.com/assets/github-b27b8e3cfb60b7141910086e009b0b884cfa3eae7920f5346168c67d274f357e.js","lineNumber":1,"columnNumber":28352},{"functionName":"n","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":10804},{"functionName":"l","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":14598},{"functionName":"p","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":15292},{"functionName":"e","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":
If you notice, the file has same thing over and over again, something is on line number 6 that is making it crash.
"functionName":"e","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":
This time, using different chrome version did not help. Since it happened with two different sites, I believe the problem is somewhere else.
same
Can you guys please check if it reproes for you on tip-of-tree puppeteer? We have a bunch of bugfixes rolled in.
Steps to reproduce
Tell us about your environment:
Chrome version tested: 59, 62, 63
What steps will reproduce the problem?
Create a script that will go to upwork, and login. Maybe first time it'll load properly, it'll fail to load on second try.
Please include code that reproduces the issue.
Works fine on Chromeless
Does not work at in puppeteer
try{ await page.goto('https://www.upwork.com/ab/account-security/login'); await page.click("#login_username") await page.type("") await page.click("#login_password") await page.type("") await page.click("#layout > div.container > div > form > div.form-group.hidden-xs.m-0-bottom > button") await page.waitFor(2000) await page.screenshot({path: 'screenshot.png'});
}catch(e){ console.log(e); await page.screenshot({path: 'screenshot.png'}); } await browser.close();