search

DevExpress / testcafe

A Node.js tool to automate end-to-end web testing.
https://testcafe.io
MIT License
9.82k stars 672 forks source link

Execute TestCafe in BrowserStack from Jenkins in Docker container stuck on "page can't be found" (initial TestCafe proxy URL) #3845

Closed marcinlesek closed 5 years ago

marcinlesek commented 5 years ago

What is your Test Scenario?

Execute tests on BrowserStack cloud, started from Jenkins job and build on Docker container. Attached configs & files below in details section.

What is the Current behavior?

When I'm starting Jenkins job, which builds Docker container which contain tests code and all dependencies and executing tests run on BrowserStack cloud - job & BS execution hangs on initial TestCafe proxy URL (page can't be found - 404), e.g.

Screenshot 2019-05-27 at 4 57 31 PM

Every another example works fine: ✅ executed test run on own host machine (MacBook Pro), ✅ executed test run on own host machine (MacBook Pro) in Docker container, ✅ executed test run on BrowserStack cloud from own host machine (MacBook Pro), ✅ executed test run on BrowserStack cloud from own host machine (MacBook Pro) in Docker container,

Executing another frameworks (e.g. WebdriverIO) in BrowserStack from Jenkins in Docker container works fine, so I assume something with TestCafe configuration must don't be properly configured? 🤔

What is the Expected behavior?

We're able to execute TestCafe test run on BrowserStack cloud from Jenkins job in Docker container.

What is your web application and your TestCafe test code?

I couldn't provide URL, but for this case, it's doesn't matter - because it couldn't connect to TestCafe.

We're using testcafe-browser-provider-browserstack to handle connection with BS.

Global envs variables ```dotenv # Browserstack BROWSERSTACK_USERNAME=... BROWSERSTACK_ACCESS_KEY=... BROWSERSTACK_USE_AUTOMATE=1 BROWSERSTACK_DISPLAY_RESOLUTION=1920x1080 BROWSERSTACK_DEBUG=true # only for this case purpose BROWSERSTACK_NETWORK_LOGS=true # only for this case purpose BROWSERSTACK_CHROME_ARGS="--no-sandbox" ```
.testcaferc.json ```json { "browsers": [ "chrome" ], "remoteChromeVersion": "74", "src": "scenarios", "reporter": [ { "name": "spec" }, { "name": "allure", "output": "reports/allure.log" }, { "name": "slack", "output": "reports/slack.log", "options": { "loggingLevel": "SUMMARY", "quietMode": true } } ], "screenshotPath": "screenshots", "takeScreenshotsOnFails": true, "screenshotPathPattern": "${DATE}_${TIME}/${TEST}/${USERAGENT}/${FILE_INDEX}.png" } ```
Tests execution script This script is used to execute test run. ```js // scripts/execute.js const createTestCafe = require('testcafe'); const args = require('commander'); const testCafeConfig = require('../.testcaferc.json'); const { browsers, remoteChromeVersion } = testCafeConfig; args .option('-b, --browserstack') .option('-s, --src ') .parse(process.argv); let testCafe = null; const getBrowserStackBrowsers = () => browsers.map((element) => { if (element === 'chrome' && remoteChromeVersion !== undefined) { return `browserstack:${element}@${remoteChromeVersion}`; } return `browserstack:${element}`; }); createTestCafe() .then((tc) => { testCafe = tc; const runner = testCafe.createRunner(); if (args.browserstack) { runner.browsers(getBrowserStackBrowsers()); } if (args.src) { runner.src(args.src); } return runner.run(); }) .then(() => { testCafe.close(); }); ```
package.json scripts ```json { "scripts": { "test": "node scripts/execute.js", "test:bs": "node scripts/execute.js --browserstack", "test:ci": "yarn test:bs" } } ```
Dockerfile ```dockerfile FROM node:10 ARG JENKINS_UID=1001 # default value, set up with build command ARG JENKINS_GID=1001 # default value, set up with build command RUN groupadd -g $JENKINS_GID jenkins && useradd -m -u $JENKINS_UID -g $JENKINS_GID jenkins ADD . /www/ RUN chown -R jenkins:jenkins /www/ ENV HOME=/home/jenkins USER jenkins WORKDIR /www RUN yarn CMD yarn test:ci ```
Execution report from Jenkins ``` [BrowserStack] BROWSERSTACK_BUILD=jenkins-tests-150 [BrowserStack] BROWSERSTACK_LOCAL=false [BrowserStack] BROWSERSTACK_BUILD=jenkins-tests-150 [BrowserStack] BROWSERSTACK_LOCAL=false [tests] $ /bin/bash -ex /tmp/jenkins5074929632456794166.sh ++ id -u ++ id -g + docker build -t tests --build-arg JENKINS_UID=107 --build-arg JENKINS_GID=112 . Sending build context to Docker daemon 3.624MB Step 1/11 : FROM node:10 ---> 5a401340b79f Step 2/11 : ARG JENKINS_UID=1000 ---> Using cache ---> 8ac3b30f4a56 Step 3/11 : ARG JENKINS_GID=1000 ---> Using cache ---> 5277c8cca05e Step 4/11 : RUN groupadd -g $JENKINS_GID jenkins && useradd -m -u $JENKINS_UID -g $JENKINS_GID jenkins ---> Using cache ---> 355985b56c83 Step 5/11 : ADD . /www/ ---> fba228938267 Step 6/11 : RUN chown -R jenkins:jenkins /www/ ---> Running in 09aa7c5dec10 Removing intermediate container 09aa7c5dec10 ---> a23d5c7991c1 Step 7/11 : ENV HOME=/home/jenkins ---> Running in 378edb74f2d9 Removing intermediate container 378edb74f2d9 ---> f30103f08dab Step 8/11 : USER jenkins ---> Running in b361ff25288b Removing intermediate container b361ff25288b ---> bd171760eaf2 Step 9/11 : WORKDIR /www ---> Running in 6531270408b4 Removing intermediate container 6531270408b4 ---> 3a4f15336e2a Step 10/11 : RUN yarn ---> Running in 3d33f74f1e6e yarn install v1.13.0 warning package.json: No license field info No lockfile found. warning package.json@1.0.0: No license field [1/4] Resolving packages... warning testcafe-browser-provider-browserstack > browserstack-local > sinon > formatio@1.1.1: This package is unmaintained. Use @sinonjs/formatio instead warning testcafe-browser-provider-browserstack > browserstack-local > sinon > samsam@1.1.2: This package has been deprecated in favour of @sinonjs/samsam warning testcafe-browser-provider-browserstack > browserstack-local > sinon > formatio > samsam@1.1.3: This package has been deprecated in favour of @sinonjs/samsam warning eslint > file-entry-cache > flat-cache > circular-json@0.3.3: CircularJSON is in maintenance only, flatted is its successor. warning testcafe > testcafe-browser-tools > babel-runtime > core-js@1.2.7: core-js@<2.6.8 is no longer maintained. Please, upgrade to core-js@3 or at least to actual version of core-js@2. [2/4] Fetching packages... [3/4] Linking dependencies... warning " > eslint-import-resolver-babel-plugin-root-import@1.1.1" has unmet peer dependency "babel-plugin-root-import@^5.1.0". [4/4] Building fresh packages... success Saved lockfile. Done in 14.65s. Removing intermediate container 3d33f74f1e6e ---> ed9667f47c20 Step 11/11 : CMD yarn test:ci ---> Running in 7d04b5c2a977 Removing intermediate container 7d04b5c2a977 ---> 631ab81fa50a Successfully built 631ab81fa50a Successfully tagged tests:latest + docker run -i --rm -v /var/lib/jenkins/workspace/tests/reports:/www/reports -v /var/lib/jenkins/workspace/tests/screenshots:/www/screenshots -e BROWSERSTACK_USERNAME=**** -e BROWSERSTACK_ACCESS_KEY=**** -e BROWSERSTACK_USE_AUTOMATE=1 -e BROWSERSTACK_DISPLAY_RESOLUTION=1920x1080 -e BROWSERSTACK_DEBUG=true -e BROWSERSTACK_NETWORK_LOGS=true -e BROWSERSTACK_CHROME_ARGS=--no-sandbox tests:latest yarn run v1.13.0 warning package.json: No license field $ yarn test:bs warning package.json: No license field $ node scripts/execute.js --browserstack The "browsers" options from the configuration file will be ignored. Running tests in: - Chrome 74.0.3729 / Mac OS X 10.14.0 (https://automate.browserstack.com/builds/xxxx/sessions/xxxx) Allure reporter started... My first fixture Build was aborted # After some time, I manually aborted the build ```
BrowserStack network logs We're achieving a lot of `404 Not Found` on basic TestCafe proxy url, BrowserStack is trying to retry this connection but without any success. ```js { pageref: "Page 0", startedDateTime: "2019-05-28T10:04:17.272-04:00", request: { method: "GET", url: "http://localhost:34165/browser/connect/dxvlU6b", httpVersion: "HTTP/1.1", cookies: [], headers: [{ name: "Host", value: "localhost:34165" }, { name: "Proxy-Connection", value: "keep-alive" }, { name: "Upgrade-Insecure-Requests", value: "1" }, { name: "User-Agent", value: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" }, { name: "Accept", value: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" }, { name: "Accept-Encoding", value: "gzip, deflate, br" }, { name: "Accept-Language", value: "en-US,en;q=0.9" } ], queryString: [], headersSize: 480, bodySize: 0, comment: "" }, response: { status: 404, statusText: "Not Found", httpVersion: "HTTP/1.1", cookies: [], headers: [{ name: "cache-control", value: "no-cache, no-store, must-revalidate" }, { name: "pragma", value: "no-cache" }, { name: "Date", value: "Tue, 28 May 2019 14:04:17 GMT" }, { name: "Connection", value: "keep-alive" }, { name: "Content-Length", value: "0" }, { name: "Proxy-Connection", value: "keep-alive" } ], content: { size: 0, mimeType: "", comment: "" }, redirectURL: "", headersSize: 208, bodySize: 0, comment: "" }, cache: {}, timings: { comment: "", send: 3, ssl: -1, wait: 311, dns: -1, connect: 1, receive: 0, blocked: -1 }, comment: "", time: 317 } ```

Steps to Reproduce:

  1. Execute Jenkins job with default hello world test in Docker container on BrowserStack with configuration from above

Your Environment details:

Does anyone experienced anything like this? I was looking for answers, trying to add "hostname": "localhost" to my .testcaferc.json, disabling BrowserStack Local etc. but without success. The most curious thing is that starting everything from my local machine, everything works.

Best regards, Marcin

AndreyBelym commented 5 years ago

Thank you for your inquiry. As far as I know, TestCafe should work the same way if you start your container from the Jenkins job. Don't try to disable BrowserStack Local, because BrowserStack cloud browsers can't connect to your machine without a tunnel created by BrowserStack Local. Try to add --hostname 127.0.0.1 --ports 9000,9001 to TestCafe parameters. To ensure that this issue is caused by TestCafe, please modify your Jenkins job to run the container from the following repository: https://github.com/AndreyBelym/browserstack-docker-example. It should produce the following build log if your Docker and Jenkins configuration is correct:

> browserstack-docker-example@1.0.0 test /app
> ./test.js

Starting tunnel...
Is Running true
Started
Hello World
marcinlesek commented 5 years ago

Thanks for fast response @AndreyBelym!

Firstly, we modify our Jenkins job to run container you linked, and it's worked correctly. Below is log:

Jenkins Job log I removed the whole build process logs, so only execution is below. ``` + docker run -i --rm -e BROWSERSTACK_USERNAME=**** -e BROWSERSTACK_ACCESS_KEY=**** browserstuck npm info it worked if it ends with ok npm info using npm@4.2.0 npm info using node@v7.10.0 npm info lifecycle browserstack-docker-example@1.0.0~pretest: browserstack-docker-example@1.0.0 npm info lifecycle browserstack-docker-example@1.0.0~test: browserstack-docker-example@1.0.0 > browserstack-docker-example@1.0.0 test /app > ./test.js Starting tunnel... Is Running true Started Most Reliable Mobile App & Browser Testing Platform | BrowserStack Stopped npm info lifecycle browserstack-docker-example@1.0.0~posttest: browserstack-docker-example@1.0.0 npm info ok Finished: SUCCESS ```

After that, I changed hostname to 127.0.0.1 and TC ports to 9000 and 9001 and below are results: ✅ executed test run on own host machine (MacBook Pro), ✅ executed test run on BrowserStack cloud from own host machine (MacBook Pro), ⛔️ executed test run on BrowserStack cloud from Jenkins job (still),

but! Now connection gone well but http authorization stopped work (my page has basic auth and I'm using .httpAuth(authCredentials) on fixture declaration) and get 401 Unauthorized in browser (from my local machine works for both cases - local and BrowserStack). What was also interesting, when BrowserStack failed and stopped execution, Jenkins still waits for response (for more than 30 minutes, after that I killed it).

So it looks like some trouble now with getting response back and http authorization. Any ideas? 🤔

AndreyBelym commented 5 years ago

It seems I forgot to push my changes to the repository. I apologize for the inconvenience, but could you please pull my changes from https://github.com/AndreyBelym/browserstack-docker-example try my example again? The SHA1 checksum of the last commit must be 3688cd7c and the last commit message must be Modify example.

marcinlesek commented 5 years ago

Hello @AndreyBelym, no problem. I re run job on Jenkins with new changes, and as I check your code and status, it hang out on closing local tunnel. I terminated job after 12 minutes. Below is log:

``` Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/AndreyBelym/browserstack-docker-example # timeout=10 Fetching upstream changes from https://github.com/AndreyBelym/browserstack-docker-example > git --version # timeout=10 > git fetch --tags --progress https://github.com/AndreyBelym/browserstack-docker-example +refs/heads/*:refs/remotes/origin/* > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > git rev-parse refs/remotes/origin/refs/heads/master^{commit} # timeout=10 Checking out Revision 3688cd7cab8ce030b31a1aaa9b0a6ae201538ded (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3688cd7cab8ce030b31a1aaa9b0a6ae201538ded Commit message: "Modify example" > git rev-list --no-walk 3688cd7cab8ce030b31a1aaa9b0a6ae201538ded # timeout=10 [BrowserStack] BROWSERSTACK_BUILD=jenkins-browserstack-test-5 [BrowserStack] BROWSERSTACK_LOCAL=false [BrowserStack] BROWSERSTACK_BUILD=jenkins-browserstack-test-5 [BrowserStack] BROWSERSTACK_LOCAL=false [browserstack-test] $ /bin/bash -ex /tmp/jenkins5297492151034805754.sh + docker build -t browserstuck . Sending build context to Docker daemon 83.97kB Step 1/6 : FROM node:10 ---> da182cf805e9 Step 2/6 : WORKDIR /app ---> Using cache ---> 7383e504b2f4 Step 3/6 : COPY package.json /app ---> Using cache ---> 328509097540 Step 4/6 : RUN ["npm", "install"] ---> Using cache ---> 58cdbd6c547b Step 5/6 : COPY . /app ---> Using cache ---> d39ba9667dcd Step 6/6 : CMD ["npm", "test"] ---> Using cache ---> 23d45cf898a7 Successfully built 23d45cf898a7 Successfully tagged browserstuck:latest + docker run -i --rm -e BROWSERSTACK_USERNAME=**** -e BROWSERSTACK_ACCESS_KEY=**** browserstuck > browserstack-docker-example@1.0.0 test /app > ./test.js Starting tunnel... Is Running true Started Hello World Build was aborted ```

In this case, it means error is on browserstack node library? 🤔

AndreyBelym commented 5 years ago

Thank you, I see that my example works in your environment with Browserstack Local enabled. Now I've disabled it in my repository. Could you please pull the changes and run my code again?

marcinlesek commented 5 years ago

Hey @AndreyBelym,

I think you forgot to remove local from https://github.com/AndreyBelym/browserstack-docker-example/blob/master/test.js#L5 and https://github.com/AndreyBelym/browserstack-docker-example/blob/master/test.js#L37 😉 

> browserstack-docker-example@1.0.0 test /app
> ./test.js

/app/test.js:5
const local = new browserstack.Local()
              ^

ReferenceError: browserstack is not defined
    at Object.<anonymous> (/app/test.js:5:15)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)
npm ERR! Test failed.  See above for more details.
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Farfurix commented 5 years ago

@marcinlesek

Hello,

Yes, you are completely right. @AndreyBelym made your changes. Could you please pull the changes and run the code again?

marcinlesek commented 5 years ago

Hello @Farfurix,

thanks for info and @AndreyBelym for update, now we'd got error from selenium-webdriver, check it below:

> browserstack-docker-example@1.0.0 test /app
> ./test.js

Starting tunnel...
Started
/app/node_modules/selenium-webdriver/lib/promise.js:2634
        throw error;
        ^

WebDriverError: [browserstack.local] is set to true but local testing through BrowserStack is not connected.
    at Object.checkLegacyResponse (/app/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/app/node_modules/selenium-webdriver/lib/http.js:509:13)
    at doSend.then.response (/app/node_modules/selenium-webdriver/lib/http.js:441:30)
    at process._tickCallback (internal/process/next_tick.js:68:7)
From: Task: WebDriver.createSession()
    at Function.createSession (/app/node_modules/selenium-webdriver/lib/webdriver.js:777:24)
    at Function.createSession (/app/node_modules/selenium-webdriver/chrome.js:719:29)
    at createDriver (/app/node_modules/selenium-webdriver/index.js:167:33)
    at Builder.build (/app/node_modules/selenium-webdriver/index.js:623:16)
    at Server.http.createServer.listen (/app/test.js:30:123)
    at Object.onceWrapper (events.js:286:20)
    at Server.emit (events.js:198:13)
    at emitListeningNT (net.js:1313:10)
    at process._tickCallback (internal/process/next_tick.js:63:19)
    at Function.Module.runMain (internal/modules/cjs/loader.js:832:11)
npm ERR! Test failed.  See above for more details.
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Farfurix commented 5 years ago

@marcinlesek

We set the browserstack.local option to false. Please try it again.

marcinlesek commented 5 years ago

Yea @Farfurix , now executed correctly, but still job hangs out. BrowserStack passed successfully, but couldn't load page that our server serves.

Screenshot 2019-06-05 at 4 19 19 PM

After 10 minutes, I terminated Jenkins job:

> browserstack-docker-example@1.0.0 test /app
> ./test.js

Starting tunnel...
Started
127.0.0.1
Stopped
Build was aborted
Aborted by Marcin Lesek
Finished: ABORTED
marcinlesek commented 5 years ago

Any ideas guys - @AndreyBelym @Farfurix @miherlosev? I'm not hiding that it's critical for us and could depend on decision to migrate from TestCafe, unfortunately. 💔

AndreyBelym commented 5 years ago

It's difficult to determine the cause of the issue without access to your infrastructure. I can only suggest you try connecting a browser from your machine to TestCafe as a remote browser when running tests with Docker and Jenkins. Likely you will need to pass the -p 9000,9001 argument to TestCafe and expose the 9000 and 9001 ports in Docker - https://docs.docker.com/docker-for-mac/networking/. If it doesn't work, you can try to start a VPN server in the container - https://medium.com/@gurayy/set-up-a-vpn-server-with-docker-in-5-minutes-a66184882c45.

marcinlesek commented 5 years ago

@AndreyBelym Solved, I managed to execute TestCafe now. Thanks for helping guys 👏

lock[bot] commented 5 years ago

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.