Acceptance tests

[individual-it]

@felixboehm :

We want to have acceptance testing for phoenix (from day one)

Here a short comparison of the proposed tools that does not claim to be complete, please add your points

Tools

Cypress https://www.cypress.io/

+ easy to install + good debugging + has a lot of knowledge about what's going on in the app/browser (more than selenium). e.g. can send raw requests, has spies, stubs etc. + cucumber plugin https://github.com/TheBrainFamily/cypress-cucumber-preprocessor - supports only Chrome* browsers https://docs.cypress.io/guides/guides/launching-browsers.html#Browsers - each test is limited to only visiting a single superdomain https://docs.cypress.io/guides/references/trade-offs.html#Same-origin (currently we are using multiple super-domains for fed. sharing. We might get around it by using multiple sub-domains) - further trade-offs https://docs.cypress.io/guides/references/trade-offs.html ? runs inside of the browser/does not use selenium (not sure if that i good or bad, a little bit of discussion here https://www.joecolantonio.com/2017/11/16/cypress-io-vs-selenium-test-automation/) ? don't really uses page objects the way we do now (that does not need to be bad, but a bit of a different approach) https://docs.cypress.io/faq/questions/using-cypress-faq.html#Can-I-use-the-Page-Object-pattern

TestCafe https://devexpress.github.io/testcafe/

+ easy to install + test debugger https://testcafe.devexpress.com/Documentation/Using_TestCafe/Test_Debugging/ + Implicit auto-waits mechanism. TestCafe automatically waits for XHR requests and page loads, so you don’t need to take care of it in your code. :man_dancing: :+1: :tada: But I wonder how good that works in practice + claims to have multi browser support https://devexpress.github.io/testcafe/documentation/using-testcafe/common-concepts/browsers/browser-support.html#officially-supported-browsers + can be used with page objects https://devexpress.github.io/testcafe/documentation/recipes/using-page-model.html#creating-a-page-model + can be used with cucumber https://github.com/helen-dikareva/testcafe-cucumber-demo ? uses a proxy between browser and server https://dzone.com/articles/testcafe-e2e-testing-tool Why are they not using selenium https://dzone.com/articles/testcafe-e2e-testing-tool ? Selenium or TestCafe https://offbeattesting.com/2018/02/02/selenium-or-testcafe/

Nightwatch.js http://nightwatchjs.org/

+ multi browser support by selenium (should be similar, with similar limitations that what we do now) + cucumber plugin https://github.com/mucsi96/nightwatch-cucumber + support for page objects http://nightwatchjs.org/guide#page-objects - problems with waiting - problems with multi browser support if browser developers do not implement the protocol correctly - basically all problems we have now with selenium ? uses selenium. We know selenium (the good, the bad and the ugly parts of it). I guess we get more or less the same what we have now with mink/selenium but in JS

CC: @felixheidecke @DeepDiver1975 @PVince81 @patrickjahns @phil-davis

[SergioBertolinSG]

There is another framework with good reviews if you want to take a look: http://webdriver.io/

[individual-it]

a POC in nightwatch.js in #133

[individual-it]

Closing this as #133 was merged