search

DevExpress / testcafe-browser-provider-saucelabs

This is the Sauce Labs browser provider plugin for TestCafe.
https://devexpress.github.io/testcafe/
MIT License
32 stars 41 forks source link

No way to use secure javascript APIs when running in saucelabs #79

Closed onlynone closed 5 months ago

onlynone commented 2 years ago

What is your Scenario?

I'd like to test a site that uses secure browser APIs like SubtleCrypto with testcafe in saucelabs.

What is the Current behavior?

I'm able to use the --hosname=localhost option as suggested here: https://testcafe.io/documentation/402839/guides/advanced-guides/test-https-features-and-http2-websites#test-https-websites in order for the browser to allow secure javascript APIs when running testcafe with local browsers. But there doesn't appear to be any way to do this when running inside saucelabs. It looks like the IP address used for the testcafe reverse-proxy in saucelabs isn't 127.0.0.1, so the --hostname=localhost trick doesn't work.

What is the Expected behavior?

There should be some way to test sites that use secure javascript APIs with testcafe and saucelabs, either through the --hostname=localhost or other options.

What is your public website URL? (or attach your complete example)

I wasn't sure where to create a PR/issue because it looks like the examples/running-tests-using-travis-and-saucelabs directory was extracted from the testcafe repo to create the testcafe-example-running-tests-using-travis-and-saucelabs repo. I created a branch/PR of that repo that demonstrates it failing:

https://github.com/DevExpress/testcafe-example-running-tests-using-travis-and-saucelabs/pull/1

The actual html being tested is:

<html>
 <head>
  <script type="text/javascript">
  function change() {
    document.getElementById("click-here").value = "Hello!";
    const a = crypto.subtle.digest("SHA-256", Uint8Array.from("test"));
  }
 </script>
 </head>
<body>
  <input type="button" value="Click here" id="click-here" onclick="change()"></input>
</body>
</html>

What is your TestCafe test code?

It's also in the referenced PR, but this is it:

import { Selector } from 'testcafe';

fixture `Check if the button text changes`
    .page `http://localhost:9090/index.html`;

test('My test', async t => {
    await t
        .click('#click-here')
        .expect(Selector('#click-here').value).eql('Hello!');
});

Your complete configuration file

N/A

Your complete test report

When run without --hostname=localhost :

$ yarn test
yarn run v1.22.17
$ testcafe "saucelabs:Chrome@beta:Windows 10" tests/index-test.js --app "node server.js"
 Running tests in:
 - Chrome 105.0.0.0 / Windows 10 (https://app.saucelabs.com/tests/...)

 Check if the button text changes
 ✖ My test

   1) A JavaScript error occurred on "http://localhost:9090/index.html".
      Repeat test actions in the browser and check the console for errors.
      To ignore client-side JavaScript errors, enable the "--skip-js-errors" CLI option, or set the "skipJsErrors" configuration file
      property to "true".
      If the website only throws this error when you test it with TestCafe, please create a new issue at:
      "https://github.com/DevExpress/testcafe/issues/new?template=bug-report.md".

      JavaScript error details:
      TypeError: Cannot read properties of undefined (reading 'digest')
          at change (http://localhost:9090/index.html:6:29)
          at HTMLInputElement.onclick (http://localhost:9090/index.html:12:185)
          at HTMLInputElement.dispatchEvent (http://192.168.1.186:57569/hammerhead.js:14:11124)
          at e._raiseDispatchEvent (http://192.168.1.186:57569/hammerhead.js:14:26617)
          at e._dispatchMouseEvent (http://192.168.1.186:57569/hammerhead.js:14:25000)
          at e._dispatchMouseRelatedEvents (http://192.168.1.186:57569/hammerhead.js:14:23729)
          at a (http://192.168.1.186:57569/hammerhead.js:14:18427)
          at e._simulateEvent (http://192.168.1.186:57569/hammerhead.js:14:18862)
          at e.click (http://192.168.1.186:57569/hammerhead.js:14:26710)
          at Ys.run (http://192.168.1.186:57569/testcafe-automation.js:1:49270)

      Browser: Chrome 105.0.0.0 / Windows 10 (https://app.saucelabs.com/tests/...)

          3 |fixture `Check if the button text changes`
          4 |    .page `http://localhost:9090/index.html`;
          5 |
          6 |test('My test', async t => {
          7 |    await t
       >  8 |        .click('#click-here')
          9 |        .expect(Selector('#click-here').value).eql('Hello!');
         10 |});
         11 |

         at <anonymous> (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/tests/index-test.js:8:10)
         at asyncGeneratorStep
      (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/tests/index-test.js:1:272)
         at _next (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/tests/index-test.js:1:610)
         at <anonymous> (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/tests/index-test.js:1:775)
         at <anonymous> (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/tests/index-test.js:1:516)
         at <anonymous> (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/tests/index-test.js:6:5)

 1/1 failed (2s)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

When run with --hostname=localhost :

$ yarn test --hostname=localhost
yarn run v1.22.17
$ testcafe "saucelabs:Chrome@beta:Windows 10" tests/index-test.js --app "node server.js" --hostname=localhost
Error: [get("http://localhost:57714/browser/connect/9Tje3tE")] Error response status: 13, UnknownError - An unknown server-side error occurred while processing the command. Selenium error: unknown error: net::ERR_CONNECTION_REFUSED
  (Session info: chrome=105.0.5195.52)
  (Driver info: chromedriver=104.0.5112.29 (eff877e18f767c77fef0481a1cba402c8cbad404-refs/branch-heads/5112@{#422}),platform=Windows NT 10.0.10586 x86_64)
    at exports.newError (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/utils.js:152:13)
    at /Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/callbacks.js:36:19
    at /Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/webdriver.js:205:5
    at Request._callback (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/http-utils.js:89:7)
    at Request.self.callback (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/node_modules/request/request.js:185:22)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:467:12)
    at Request.<anonymous> (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:467:12)
ERROR Unable to open the "saucelabs:Chrome@beta:Windows 10" browser due to the following error:

Error: [get("http://localhost:57714/browser/connect/9Tje3tE")] Error response status: 13, UnknownError - An unknown server-side error occurred while processing the command. Selenium error: unknown error: net::ERR_CONNECTION_REFUSED
  (Session info: chrome=105.0.5195.52)
  (Driver info: chromedriver=104.0.5112.29 (eff877e18f767c77fef0481a1cba402c8cbad404-refs/branch-heads/5112@{#422}),platform=Windows NT 10.0.10586 x86_64)
    at exports.newError (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/utils.js:152:13)
    at /Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/callbacks.js:36:19
    at /Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/webdriver.js:205:5
    at Request._callback (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/lib/http-utils.js:89:7)
    at Request.self.callback (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/node_modules/request/request.js:185:22)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:467:12)
    at Request.<anonymous> (/Users/steven/projects/testcafe-example-running-tests-using-travis-and-saucelabs/node_modules/wd/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:467:12)

Type "testcafe -h" for help.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Screenshots

No response

Steps to Reproduce

  1. Checkout the branch of the testcafe-example-running-tests-using-travis-and-saucelabs repo referenced in the PR mentioned above
  2. Run yarn test
  3. Run yarn test --hostname=localhost

TestCafe version

1.20.1

Node.js version

v14.15.4

Command-line arguments

testcafe "saucelabs:Chrome@beta:Windows 10" tests/index-test.js --app "node server.js" [--hostname=localhost]

Browser name(s) and version(s)

chrome=105.0.5195.52

Platform(s) and version(s)

Windows 10

Other

No response

onlynone commented 2 years ago

And just for reference, if you run the exact same code, but with a local browser not through saucelabs with: yarn testcafe chrome tests/index-test.js --app "node server.js" --hostname=localhost it succeeds.

miherlosev commented 2 years ago

Hi @onlynone

Could you please share the server.js file?

NeoPhi commented 2 years ago

@miherlosev The PR was a clone of testcafe-example-running-tests-using-travis-and-saucelabs so it is using the stock server.js from that project, a copy of which can be found here: https://github.com/onlynone/testcafe-example-running-tests-using-travis-and-saucelabs/blob/secure_apis/server.js

miherlosev commented 2 years ago

Hi, @onlynone and @NeoPhi Thank you for the shared information. I've reproduced the issue.

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

github-actions[bot] commented 5 months ago

We're closing this issue after a prolonged period of inactivity. If it still affects you, please add a comment to this issue with up-to-date information. Thank you.