Consider adding real e2e tests

[sobolevn]

This looks promising: https://gitlab.com/cypress-io/cypress-example-docker-gitlab

Working example: https://github.com/chrisvfritz/vue-enterprise-boilerplate/tree/master/tests/e2e Docs: https://github.com/chrisvfritz/vue-enterprise-boilerplate/blob/master/docs/tests.md#end-to-end-tests-with-cypress

[sobolevn]

Limitations that really bother me:

• https://github.com/cypress-io/cypress/issues/310
• speed, it very slow to run on every PR and after PR is merged
• docker support, it does not support alpine, it also misses any node-version support. So we will have to sync node version by ourself

[sobolevn]

One more example: https://github.com/chymz/nuxt-template/tree/master/template/cypress

[sobolevn]

Ok, here's my thoughts about this problem:

Framework	Engine	Assertions	Different browsers	Speed	Docker	Linting	User interactions
Cypress	Tests are injected into the web page	Custom	Some	Bearable	Slim	Custom eslint config	Yes
JSDom	Native server-side JS	Any	No	Best	Any, including Alpine	Not needed	No
Nightwatch	Selenium/WebDriver	Custom	Yes	Slow	Container Hell	Not available	Yes
Puppeteer	Headless Chrome	Any	No	Bearable	Alpine	Not available	Yes
TestCafe	Tests are injected into the web page	Custom	Yes	Bearable	Alpine	Custom eslint config	Yes

Features:

Resources

Resources used in this research:

• https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
• https://quizlet.com/blog/cypress-the-future-of-end-to-end-testing-for-web-applications
• https://medium.com/@adrian_lewis/top-5-most-rated-node-js-frameworks-for-end-to-end-web-testing-f8ebca4e5d44
• https://medium.com/tech-quizlet/cypress-the-future-of-end-to-end-testing-for-web-applications-8ee108c5b255

Showcase

Cypress: https://github.com/chrisvfritz/vue-enterprise-boilerplate/tree/master/tests/e2e TestCafe: https://github.com/egoist/vuepack/blob/master/template/test/e2e/index.js Puppeteer: https://github.com/studbits/nuxt-jest-puppeteer

[sobolevn]

TestCafe

https://github.com/DevExpress/testcafe#creating-the-test

Sample:

import { Selector } from 'testcafe'; // first import testcafe selectors

fixture `Getting Started`// declare the fixture
    .page `https://devexpress.github.io/testcafe/example`;  // specify the start page

//then create a test and place your code there
test('My first test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button')

        // Use the assertion to check if the actual header text is equal to the expected one
        .expect(Selector('#article-header').innerText).eql('Thank you, John Smith!');
});

Running: npx testcafe chrome testcafe.js

Result:

» npx testcafe chrome testcafe.js 
npx: installed 285 in 21.6s
 Running tests in:
 - Chrome 63.0.3239 / Mac OS X 10.11.6

 Getting Started
 ✓ My first test

 1 passed (3s)

Features

• flow support: https://github.com/DevExpress/testcafe/blob/master/docs/articles/documentation/recipes/finding-code-issues-with-flow-type-checker.md
• vscode debug: https://github.com/DevExpress/testcafe/blob/master/docs/articles/documentation/recipes/debugging-with-visual-studio-code.md
• vue selector: https://github.com/DevExpress/testcafe-vue-selectors
• nuxt selector: https://github.com/kartojal/testcafe-nuxt-selectors
• browserstack integration: https://github.com/DevExpress/testcafe-browser-provider-browserstack

[sobolevn]

TestCafe + Nuxt

Sample:

import VueSelector from 'testcafe-nuxt-selectors'

fixture('Getting Started')
  .page('http://localhost:3000')
  .beforeEach(async () => await VueSelector())

test('My first test', async (t) => {
  const root = VueSelector()
  const rootVue = await root.getVue()

  await t.expect(root.exists).ok()
  await t.expect(rootVue.state.layoutName).eql('default')
})

Output:

» ./node_modules/.bin/testcafe chrome  --app "yarn dev" --app-init-delay 10000 testcafe.js
 Running tests in:
 - Chrome 63.0.3239 / Mac OS X 10.11.6

 Getting Started
 ✓ My first test

 1 passed (2s)

Issues

• Nuxt is not working properly https://github.com/DevExpress/testcafe-vue-selectors/issues/25
• Working with this fix: return (instance.$options._componentTag || instance.$options.__file || '')
• Nuxt and Vue won't work in production mode? That's quite sad. I don't want to test a development version

Testing production version

I have implemented this kind of pipeline: nuxt build && ./node_modules/.bin/testcafe chrome --app "yarn start" --app-init-delay 10000 testcafe.js

This way it works. Mostly.

[christian-bromann]

@sobolevn why not consider webdriver.io?

[sobolevn]

@christian-bromann thanks for advice! I did not know that this product exists.

I try to remove selenium from our toolchain. The main reason for it is speed and setup complexity. We run everything inside docker-compose for development and testing. And I need a simple solution to just work in this setup. When we had nightwatch with webdriver bindings it was really hard to manage all the test containers with jvm and stuff.

Speed. I need to run all tests before merging a PR. So, it should not take long to start and execute tests. My previous experience with selenium was too slow.

Can you please correct me if I am wrong about these two point? Thanks!

[christian-bromann]

You can definitely get a decent speed when setting up the driver properly. By running Chrome headless with Chromedriver (instead of the Selenium standalone server) your test should be quick. A quick google search gave me this project which seems to exactly do that: https://github.com/kaitoy/webdriverio-chrome

[kartojal]

Just to contribute with my grain of sand, you can run Testcafe with concurrency enabled, with chrome and/or firefox in headless mode. For example you can divide 100 tests into 2 concurrent headless chrome browsers, doing each instance 50 tests. In that way it improves the testing speed. Ex: testcafe chrome:headless -c 2 test/e2e/*spec*

[sobolevn]

@kartojal thanks! Currently testing it out.