puppeteer / puppeteer

JavaScript API for Chrome and Firefox
https://pptr.dev
Apache License 2.0
88.37k stars 9.07k forks source link

Login on Upwork/Github crashes chrome #927

Closed entrptaher closed 6 years ago

entrptaher commented 7 years ago

Steps to reproduce

Tell us about your environment:

Chrome version tested: 59, 62, 63

What steps will reproduce the problem?

Create a script that will go to upwork, and login. Maybe first time it'll load properly, it'll fail to load on second try.

Please include code that reproduces the issue.

  1. Works fine on Chromeless

    const chromeless = new Chromeless({ remote: true })
    const screenshot = await chromeless
    .goto('https://www.upwork.com/ab/account-security/login')
    .type("*********", "#login_username")
    .type("*********","#login_password")
    .click("#layout > div.container > div > form > div.form-group.hidden-xs.m-0-bottom > button")
    .wait(20000)
    .screenshot()
    console.log(screenshot)
    await chromeless.end()
  2. Does not work at in puppeteer

    
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

try{ await page.goto('https://www.upwork.com/ab/account-security/login'); await page.click("#login_username") await page.type("") await page.click("#login_password") await page.type("") await page.click("#layout > div.container > div > form > div.form-group.hidden-xs.m-0-bottom > button") await page.waitFor(2000) await page.screenshot({path: 'screenshot.png'});

}catch(e){ console.log(e); await page.screenshot({path: 'screenshot.png'}); } await browser.close();



**What is the expected result?**
Disclaimer: The account used was a dummy account.

The page will be logged in nicely.
![image](https://user-images.githubusercontent.com/8284972/31058199-9ea9ab7e-a711-11e7-95a4-84c2bb6b0b9a.png)
![image](https://user-images.githubusercontent.com/8284972/31058229-0a90f8b0-a712-11e7-9be3-0927b278153b.png)

**What happens instead?**
The page does not load properly, let alone login.
![image](https://user-images.githubusercontent.com/8284972/31058201-a408c92e-a711-11e7-9a16-9829a25ecf97.png)

![image](https://user-images.githubusercontent.com/8284972/31058228-008982ba-a712-11e7-890b-d41d5769433e.png)

Additional Info:
- Did you try it in local?
Yes. Same result.

- Did you try to use proxies?
Yes. Same result.

- Does it happen if you go to that page normally with your browser?
No. The page loads normally.

- Why compare to chromeless?
That is using chromium and passing the problem pretty easy, while puppeteer is not. Notice how recaptcha is screaming about upgrading browser on puppeteer.
ebidel commented 6 years ago

So the issue is that headless is triggering the capchta page? What happens if you launch with full chrome?

On Sun, Oct 1, 2017, 9:39 AM Abu Taher notifications@github.com wrote:

Steps to reproduce

Tell us about your environment:

What steps will reproduce the problem?

Create a script that will go to upwork, and login. Maybe first time it'll load properly, it'll fail to load on second try.

Please include code that reproduces the issue.

  1. Works fine on Chromeless

const chromeless = new Chromeless({ remote: true })const screenshot = await chromeless .goto('https://www.upwork.com/ab/account-security/login') .type("", "#login_username") .type("","#login_password") .click("#layout > div.container > div > form > div.form-group.hidden-xs.m-0-bottom > button") .wait(20000) .screenshot()console.log(screenshot)await chromeless.end()

  1. Does not work at in puppeteer

const browser = await puppeteer.launch();const page = await browser.newPage(); try{await page.goto('https://www.upwork.com/ab/account-security/login');await page.click("#login_username")await page.type("")await page.click("#login_password")await page.type("")await page.click("#layout > div.container > div > form > div.form-group.hidden-xs.m-0-bottom > button")await page.waitFor(2000) await page.screenshot({path: 'screenshot.png'});

}catch(e){ console.log(e); await page.screenshot({path: 'screenshot.png'}); }await browser.close();

What is the expected result? Disclaimer: The account used was a dummy account.

The page will be logged in nicely. [image: image] https://user-images.githubusercontent.com/8284972/31058199-9ea9ab7e-a711-11e7-95a4-84c2bb6b0b9a.png [image: image] https://user-images.githubusercontent.com/8284972/31058229-0a90f8b0-a712-11e7-9be3-0927b278153b.png

What happens instead? The page does not load properly, let alone login. [image: image] https://user-images.githubusercontent.com/8284972/31058201-a408c92e-a711-11e7-9a16-9829a25ecf97.png

[image: image] https://user-images.githubusercontent.com/8284972/31058228-008982ba-a712-11e7-890b-d41d5769433e.png

Additional Info:

-

Did you try it in local? Yes. Same result.

Did you try to use proxies? Yes. Same result.

Does it happen if you go to that page normally with your browser? No. The page loads normally.

Why compare to chromeless? That is using chromium and passing the problem pretty easy, while puppeteer is not. Notice how recaptcha is screaming about upgrading browser on puppeteer.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/puppeteer/issues/927, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOigFvbbV7B_7Yjx0D1oyOw6SWBF-Ocks5sn-p1gaJpZM4PqARB .

entrptaher commented 6 years ago

If I used my own chrome/firefox, no problem happens.

ebidel commented 6 years ago

That's what I suspected. I've seen other sites think headless Chrome (and using it in browser automation) as a bot. We've seen that over in Lighthouse too.

Captchas tend to trigger on all sorts of things (suspect mouse movement, suspect browsing history, etc.) Unfortunately, they don't give us much more than that for good reason :p

entrptaher commented 6 years ago

If I normally use puppeteer {headless:false}, I made sure to use different proxies, without proxies, with proxies, etc etc.

  1. captcha happens after I bypass login page.
  2. Or now it's worse, the whole chromium crashes.
Received signal 11 SEGV_MAPERR 000000000020
#0 0x564d00d2c337 base::debug::StackTrace::StackTrace()
#1 0x564d00d2beaf base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7fa49d605390 <unknown>
#3 0x564cfffc2cbe device::PlatformSensorProviderBase::MapSharedBufferForType()
#4 0x564cfffc7803 device::PlatformSensorProviderLinux::CreateSensorAndNotify()
#5 0x564cfffc77aa device::PlatformSensorProviderLinux::ProcessStoredRequests()
#6 0x564d00d2ca6b base::debug::TaskAnnotator::RunTask()
#7 0x564d00d447fb base::MessageLoop::RunTask()
#8 0x564d00d44e6b base::MessageLoop::DoWork()
#9 0x564d00d46b79 base::MessagePumpLibevent::Run()
#10 0x564d00d63210 base::RunLoop::Run()
#11 0x564cffbc5996 content::BrowserThreadImpl::IOThreadRun()
#12 0x564cffbc5b91 content::BrowserThreadImpl::Run()
#13 0x564d00d87d09 base::Thread::ThreadMain()
#14 0x564d00d82a83 base::(anonymous namespace)::ThreadFunc()
#15 0x7fa49d5fb6ba start_thread
#16 0x7fa49735d3dd clone
  r8: 0000000000000000  r9: 0000000000000000 r10: 0000000000000000 r11: 000000000000063c
 r12: 0000000000000c10 r13: 00007fa480dca6b8 r14: 00007fa480dca398 r15: 0000000000000005
  di: 00007fa480dca1a8  si: 0000000000000001  bp: 0000000000000000  bx: 000001fb3ff74398
  dx: 00000000000001c0  ax: 0000000000000000  cx: 0000000000000000  sp: 00007fa480dca370
  ip: 0000564cfffc2cbe efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000006
 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000020
[end of stack trace]
Calling _exit(1). Core file will not be generated.

And I searched for the crash, I found it's a chromium bug introduced after 59th version. Still, I wonder, why specific sites always makes the problem? and why only chromeless does not have the problem? and why only nightmare and puppeteer triggers the bot/crash etc.

ebidel commented 6 years ago

Can you post a link to the crbug?

entrptaher commented 6 years ago

I followed along the versions, tried version 58, which does not have headless. Obviously it did not work. I was having problem with version 59-62 and I saw version 63. The crash is no more if I use the latest Version 63.0.3236.0 (Developer Build). Nor do I see the captcha. Also I am using the following arguments, the disable extensions part was suggested on various replies but it did not work on the mentioned versions above.

'--enable-logging --v=1', '--ignore-certificate-errors', '--disable-extensions'

image

Bug report and other related Links for research purpose in case someone/me has this problem in future:

And, more debugging gave me this right before the crash,

Sat, 07 Oct 2017 09:02:44 GMT puppeteer:session ◀ RECV {"method":"Network.requestWillBeSent","params":{"requestId":"14756.35","loaderId":"14756.2","documentURL":"https://www.upwork.com/ab/account-security/login","request":{"url":"https://collector-pxss13u803.perimeterx.net/api/v1/collector","method":"POST","headers":{"Referer":"https://www.upwork.com/ab/account-security/login","Origin":"https://www.upwork.com","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3198.0 Safari/537.36","Content-type":"application/x-www-form-urlencoded"},"postData":"payload=W3sidCI6IlBYMiIsImQiOnsiUFg2MyI6IkxpbnV4IHg4Nl82NCIsIlBYOTYiOiJodHRwczovL3d3dy51cHdvcmsuY29tL2FiL2FjY291bnQtc2VjdXJpdHkvbG9naW4ifX1d&appId=PXSs13U803&tag=v2.48&uuid=47de4300-ab3e-11e7-b5b9-7728bb3e0aff","mixedContentType":"none","initialPriority":"High","referrerPolicy":"no-referrer-when-downgrade"},"timestamp":25559.40582,"wallTime":1507366964.03026,"initiator":{"type":"script","stack":{"callFrames":[{"functionName":"nrWrapper","scriptId":"31","url":"https://www.upwork.com/ab/account-security/login","lineNumber":6,"columnNumber":11614},{"functionName":"_postXhr","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":14858},{"functionName":"sendActivities","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":14247},{"functionName":"sendClientActivities","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":10212},{"functionName":"bootstrap","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":9907},{"functionName":"Vn","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":10771},{"functionName":"","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":23785},{"functionName":"","scriptId":"83","url":"https://client.perimeterx.net/PXSs13U803/main.min.js","lineNumber":2,"columnNumber":23790}]}},"type":"XHR","frameId":"14756.1"}}
Received signal 11 SEGV_MAPERR 000000000020
#0 0x564911925337 base::debug::StackTrace::StackTrace()
#1 0x564911924eaf base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7f91a8c3a390 <unknown>
#3 0x564910bbbcbe device::PlatformSensorProviderBase::MapSharedBufferForType()
#4 0x564910bc0803 device::PlatformSensorProviderLinux::CreateSensorAndNotify()
#5 0x564910bc07aa device::PlatformSensorProviderLinux::ProcessStoredRequests()
#6 0x564911925a6b base::debug::TaskAnnotator::RunTask()
#7 0x56491193d7fb base::MessageLoop::RunTask()
#8 0x56491193de6b base::MessageLoop::DoWork()
#9 0x56491193fb79 base::MessagePumpLibevent::Run()
#10 0x56491195c210 base::RunLoop::Run()
#11 0x5649107be996 content::BrowserThreadImpl::IOThreadRun()
#12 0x5649107beb91 content::BrowserThreadImpl::Run()
#13 0x564911980d09 base::Thread::ThreadMain()
#14 0x56491197ba83 base::(anonymous namespace)::ThreadFunc()
#15 0x7f91a8c306ba start_thread
#16 0x7f91a29923dd clone
  r8: 0000000000000000  r9: 0000000000000000 r10: 0000000000000000 r11: 000000000000063c
 r12: 0000000000000c10 r13: 00007f918c3ff6b8 r14: 00007f918c3ff398 r15: 0000000000000005
  di: 00007f918c3ff1a8  si: 0000000000000001  bp: 0000000000000000  bx: 00003e1ecf3511e8
  dx: 00000000000001c0  ax: 0000000000000000  cx: 0000000000000000  sp: 00007f918c3ff370
  ip: 0000564910bbbcbe efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000006
 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000020
[end of stack trace]

The script it's trying to load is https://client.perimeterx.net/PXSs13U803/main.min.js, which has bunch of PDF and Canvas related stuff.

If I did this,

await page.setRequestInterceptionEnabled(true);
          page.on('request', request => {
            if (request.url.indexOf('perimeterx') > -1) {
              console.log(`request blocked`);
              request.abort();
            } else {
              request.continue();
            }
          });

image

then, the page does not crash. But it's not like I can monitor for one url out of thousands all the time. and it's not like I can mention which chrome version to download when I install puppeteer...?

Maybe some of these triggers the crash?

var n = {};
      n.PX31 = e,
      n.PX32 = t;
      var r = s(),
        o = eo(),
        a = Kr();
      n.PX274 = a.canvasData,
      n.PX275 = a.canvasWinding,
      n.PX276 = o.canvasfp,
      n.PX210 = o.webglRenderer,
      n.PX209 = o.webglVendor,
      n.PX277 = o.webGLVersion,
      n.PX278 = o.shadingLangulageVersion,
      n.PX279 = o.unmaskedVendor,
      n.PX280 = o.unmaskedRenderer,
      n.PX281 = o.extensions,
      n.PX282 = o.webglParameters,
      n.PX33 = s() - r,
      n.PX168 = De(),
      n.PX248 = We(window.document),
      n.PX249 = We(window),
      n.PX57 = i(),
      n.PX264 = He(),
      n.PX265 = je(),
      n.PX266 = Ve(window),
      n.PX89 = Ue(),
      ue(n, "PX286", function() {
        return window.devicePixelRatio || ""
      }, ""),
      ue(n, "PX287", function() {
        return navigator.hardwareConcurrency
      }, -1),
      ue(n, "PX288", function() {
        return !!window.localStorage
      }, !1),
      ue(n, "PX289", function() {
        return !!window.indexedDB
      }, !1),
      ue(n, "PX290", function() {
        return !!window.openDatabase
      }, !1),
      ue(n, "PX291", function() {
        return !!document.body.addBehavior
      }, !1),
      ue(n, "PX292", function() {
        return navigator.cpuClass
      }),
      ue(n, "PX293", function() {
        return !!window.sessionStorage
      }, !1);
entrptaher commented 6 years ago

I found similar issue on other login pages too. For example I tried to login to github using puppeteer and got same stack error. The normal browser works fine, so I believe something is missing here.

This is a related bug with similar stack trace: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1702501

And same with upwork, it fails after this line,

Sat, 21 Oct 2017 17:28:10 GMT puppeteer:session ◀ RECV {"method":"Network.requestWillBeSent","params":{"requestId":"4549.37","loaderId":"4549.4","documentURL":"https://github.com/","request":{"url":"https://collector.githubapp.com/collect","method":"POST","headers":{"Cache-Control":"max-age=0","Origin":"https://github.com","Referer":"https://github.com/","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.79 Chrome/61.0.3163.79 Safari/537.36","Content-Type":"application/x-www-form-urlencoded"},"postData":"","mixedContentType":"none","initialPriority":"VeryLow","referrerPolicy":"origin-when-cross-origin"},"timestamp":101881.974186,"wallTime":1508606890.56355,"initiator":{"type":"script","stack":{"callFrames":[{"functionName":"a","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":1,"columnNumber":31058},{"functionName":"","scriptId":"38","url":"https://assets-cdn.github.com/assets/github-b27b8e3cfb60b7141910086e009b0b884cfa3eae7920f5346168c67d274f357e.js","lineNumber":1,"columnNumber":24857},{"functionName":"ie.D","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":13033},{"functionName":"In.send","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":28173},{"functionName":"n.(anonymous function)","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":6607},{"functionName":"rr.v","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":145},{"functionName":"rr.D","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":5,"columnNumber":31648},{"functionName":"or","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":200},{"functionName":"u","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":5484},{"functionName":"","scriptId":"38","url":"https://assets-cdn.github.com/assets/github-b27b8e3cfb60b7141910086e009b0b884cfa3eae7920f5346168c67d274f357e.js","lineNumber":1,"columnNumber":28352},{"functionName":"n","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":10804},{"functionName":"l","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":14598},{"functionName":"p","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":15292},{"functionName":"e","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":

If you notice, the file has same thing over and over again, something is on line number 6 that is making it crash.

"functionName":"e","scriptId":"37","url":"https://assets-cdn.github.com/assets/frameworks-39e27a918ffc2ea974ee623d9cea7a65213f4945201fadbade247e4792da6fb4.js","lineNumber":6,"columnNumber":

This time, using different chrome version did not help. Since it happened with two different sites, I believe the problem is somewhere else.

Textras commented 6 years ago

same

aslushnikov commented 6 years ago

Can you guys please check if it reproes for you on tip-of-tree puppeteer? We have a bunch of bugfixes rolled in.