search

DevExpress / testcafe-browser-provider-saucelabs

This is the Sauce Labs browser provider plugin for TestCafe.
https://devexpress.github.io/testcafe/
MIT License
32 stars 41 forks source link

Sporadic saucelabs failures Error: [eval("")] Error response status: 17 JavaScriptError #64

Closed vire closed 4 years ago

vire commented 4 years ago

This failure appears sporadically on CI and locally

node v14.7.0
yarn 1.22.4
testcafe": "1.9.1
testcafe-browser-provider-saucelabs@1.8.0
wd@1.11.4

image

with testcafe": "1.9.1 the problem was not present

I have a suspicion it's related to the sauce connect tunnel...

Any ideas for debugging?

AndreyBelym commented 4 years ago

Thank you for your report. Can you reproduce the issue with a local browser? Since you mentioned that the issue didn't exist in the previous TestCafe version, I think it is not Saucelabs specific. Could you please try previous TestCafe versions and find where the issue started to appear?

It's also strange that the wd error mentions Chrome 83, while TestCafe mentions Chrome 78. Could you please add a full TestCafe command line options and the configuration file and a full test report?

Also, it will be great if you can share a sample test and a page that can be used to reproduce the problem.

vire commented 4 years ago

@AndreyBelym I'll try to create s OSS repo replicating the problem but I'll do it not sooner than over weekend...

For now - I've reverted back to

testcafe": "1.6.1
testcafe-browser-provider-saucelabs@1.7.0

but have changed the browsers definition as following

image

And I get the same error - when I don't modify browsers I get hanging timeout - I'm lacking here a bit of some progress maybe a debugging mode to see how the communication with saucelabs works

AndreyBelym commented 4 years ago

Do I understand correctly that with the latest browsers TestCafe v1.6.1 works fine? Does TestCafe v1.9.1 also work fine with the latest browsers?

Please note that TestCafe supports the latest browser versions officially. Some very old browser versions may not function properly. Edge 15 is heavily outdated and is not supported by TestCafe: https://github.com/DevExpress/testcafe/issues/5302#issuecomment-656567262

I don't think the error in the starting post relates to the outdated Edge somehow. This is an uncaught JS exception that occurs on your page. Without Saucelabs, these errors are captured by TestCafe and lead to test failures until you use --skip-js-errors switch. Do you use the -e or --skip-js-errors switches, could you please share the full TestCafe configuration and command-line arguments? Did you try to reproduce the issue with a local browser?

I look forward to hearing from you so that we could resolve the issue and get it right.

vire commented 4 years ago

@AndreyBelym I had no time to try this out - I think the issue is with my setup, or with our connection to saucelabs, or reaching docker from saucelabs etc. If no one else had this issue of spontaneous hangs of the testcafe-browser-provider-saucelabs then it mostly will be our issue.

Closing until I get some proper results