Automate Sauce Lab browser tests in Travis for PRs

[cfarm]

• Browser acceptance tests for cfgov-refresh run on every PR - we could run these on all supported browsers in Sauce
• Use the script from cfpb-chart-builder as a basis https://github.com/cfpb/cfpb-chart-builder/blob/master/test/browser-tests.js

[cfarm]

• [x] Get Sauce browser tests running locally in cfgov-refresh @virginiacc @cfarm

[cfarm]

short-term goal: run browser tests with Sauce Labs on push using Travis; get at least one browser working (latest IE) and see how this affects the time for builds

[Scotchester]

• [ ] @contolini and I to work on remembering how to write browser tests

[cfarm]

@Scotchester and document it ✍️ 📓 📝

[cfarm]

Notes from Virginia's and my pairing session 7/19 testing Sauce Labs acceptance tests locally

Questions

• Do we need a SAUCE_TUNNEL env variable? -
  • Answer: This can be a string you can use to name your tests.
• Can we pass in a browser/platform combo to override the defaults? Code appears to be set up this way
• Can we run with just the legacy or modern set of browsers from default-suites.js?
  • Unclear
• How long does it take to run all the default suite browsers?
  • 41 minutes!!
• How long does it take to run one or two browsers?
  • 7 minutes to run one browser
• What does the --suite param do? "--suite: Choose a particular suite or suites to run."
  • Unclear. It does NOT let you run a different suite of browsers, which is what we thought it must be.

Todos

• [ ] Figure out and document how to run legacy or modern set of browsers from default-suites.js
• [ ] Update testing doc to include SAUCE_TUNNEL id value and explain it @cfarm
  • Document this! It's required. It can be a name, it appears in the SauceLabs.com Dashboard. We should provide a default in the ENV_SAMPLE file and in the documentation.
• [x] Add how to test individual browsers and override default suite. There is some info in manual testing section already. Example command: @virginiacc gulp test:acceptance --sauce --browserName='internet explorer' --version='9' --platform='Windows 7'
• [x] Can we run these tests against localhost or build using the Sauce tunnel - yes you have to use gulp test:acceptance:protractor --sauce=true note the --sauce flag alone doesn't work. you have to have a local server running.
• [x] Can we run these against live site URL? yes with some tweaks to the environment-test.js and .env file
• [ ] Update environment-test.js so we can run with http OR https in the URL for localhost URL @cfarm, so you can run the browser tests when using HTTPS locally with ./runserver.sh ssl
• [x] Update environment-test.js so we can run www.consumerfinance.gov without the PORT constant in our .env @virginiacc (just need to set DJANGO_HTTP_PORT='' in .env; the test env var TEST_HTTP_PORT maps to DJANGO_HTTP_PORT, which is set to 8000 in runserver.sh and needs to be overridden for local testing against production)
• [ ] Rewrite the existing tests to run without tox? All of FEWDs can participate and divvy these up

Improvements

• [ ] Fix all the failing tests, so we can know when we've introduced a new failure.
• [ ] Add a mobile test suite of mobile device browsers to default-suites.js
• [ ] Fix this bug - When you run the sauce tests without a local server running and without tox with gulp test:acceptance:protractor --sauce=true --browserName='internet explorer' --version='11' --platform='Windows 7', all the tests should fail. But some of them unexpectedly pass.
• [ ] Automate running all the browser tests in off-hours using Travis, so they don't block the queue and so we can keep an eye on when we've introduced new failures. https://docs.travis-ci.com/user/cron-jobs/

[virginiacc]

Test suite failures on Sauce labs

Scenario	Failure	Browser
Large Size, when mouse is over link (mega-menu.feature:13)	✖ Then the mega-menu organism shouldn't show the first link immediately	internet explorer 10.0 Windows 7 31-4 internet explorer Windows 10 21-4 chrome Windows 01
Large Size, when mouse is over link (mega-menu.feature:13)	✖ Then the mega-menu organism shouldn't show the first link immediately (UnsupportedOperationError: mouseMoveTo)	firefox Windows 10 11-4
State on page load (multi-select.feature:10)	✖ Then the multi-select should be rendered (multi-select.js:63)	internet explorer Windows 10
Navigate to the previous page (pagination.feature:14)	✖ Then the page url should contain "page=1"	internet explorer 10.0 Windows 7
Navigate to nth Page (pagination.feature:19)	✖ Then the page url should contain "page=2"	internet explorer Windows 10
Large Size, should navigate to search portal (global-search.feature:24)	✖ Then I should navigate to search portal	internet explorer 10.0 Windows 7

[cfarm]

Do we ever run all the browser tests in any environment, such as Jenkins? I'm going to dig around docs/JAC jobs to find out but if anyone knows the answer let us know. From what I remember when talking about it in the past, the answer was 'no.'

[chosak]

@cfarm afaik browser tests are never run on Jenkins, only on Travis.

[cfarm]

OK. The browser tests are run in Chrome with Travis, so it seems that none of the other browsers are tested automatically: https://github.com/cfpb/cfgov-refresh/blob/master/test/browser_tests/default-suites.js#L64

[cfarm]

For scheduling all the browser tests to run outside of the Travis PR queue, we looked at https://circleci.com/docs/2.0/workflows/#scheduling-a-workflow which allows us to schedule a specific time to run the browser tests every day. This wouldn't block our PRs since it'd be running through CircleCI instead of Travis. We could also use CircleCI eventually to run performance tests, visual regression tests, etc - other tests that we normally don't implement on Travis because we worry about blocking PRs.

Adam would have to approve CircleCI app on the cfpb organization for us to try this out.

Edited to add: can we also set up Sauce Labs to run these as a cron job? Something to investigate

[cfarm]

Plan for next Thursday 3/ 👍

• Virginia and Cat will pair looking at the test failures in IE10 to figure out what's causing them. Anything that's failed because of a JS feature, we would want to remove that test from running on IE 10.

Future discussions:

• Browser test strategy. We're not writing new browser tests, or updating them regularly. And there are unit tests that would be better as browser tests.
• Run tests against live site on a regular basis, too.

[virginiacc]

Travis cron jobs

We initially looked into the possibility of creating a Travis cron job to run the browser tests in Sauce at night so it wouldn't impact our Travis queue, but it appears that Travis only offers the option to set an interval (daily, weekly, monthly) for a cron job, not a specific start time. It does sound like jobs run at approximately the same time you first set them up, so a workaround could be to initially enable the job at night. Using something like CircleCI to completely bypass the Travis queue might be a cleaner option, though.

[cfarm]

Here's my WIP branch with changes to the browser tests documentation...added info on how to run tests against localhost, consumerfinance.gov, etc. I haven't fully tested some of the instructions I wrote here, so they are probably wrong in places (it's based on piecing together memories of last July's findings) https://github.com/cfpb/cfgov-refresh/pull/4859

[cfarm]

Cron jobs follow-up

Travis and CircleCI cron jobs for the cross-browser tests are now on hold. We may be able to use CircleCI, we need to talk to @marcesher about our use case and get a PUA going to be able to enable it on cfgov-refresh.

For now, we will continue our proof-of-concept for running cross-browser tests on Sauce Labs by using Jenkins - for reference we will use this technique https://github.com/cfpb/cfgov-refresh/pull/new/new-acceptance-tests-feedback

[contolini]

JAC the whats-in-a-name job to use node > 8