Determine our end-to-end (E2E) testing strategy

[patcon]

Re-ticketed from https://github.com/pol-is/polis-issues/issues/127#issuecomment-621391419

End-to-end tests are tests that run against the whole app by manipulating the user interface, using tools like Selenium or Puppeteer.

Summary

• @crkrenn and @david-nadaraia wrote some Puppeteer tests
• @ballPointPenguin has had good experience with Cypress (about)
• was decided on last month's call that writing tests is favoured by all of us
• @patcon has no tool pref, and is ok using multiple tools for starters
• @ballPointPenguin offered to work on PoC for cypress the week of 2020-05-01

[patcon]

From https://github.com/pol-is/polis-issues/issues/127#issuecomment-621391419

@ballPointPenguin:

@crkrenn @david-nadaraia how invested in Puppeteer are you at this point? The reason I ask is that I'm a big fan of Cypress and would like that to be considered. I realize I should have chimed in earlier, so if the Puppeteer ship is already sailing, I will join the crew. I have set up Cypress testing for end-to-end javascript heavy web applications at a few companies now and as a veteran web app tester, I am extremely impressed. Here's a poignant medium article if you want to read more https://medium.com/tech-quizlet/cypress-the-future-of-end-to-end-testing-for-web-applications-8ee108c5b255 which makes comparison to Puppeteer and others.

@crkenn:

David wrote a few tests with Puppeteer

https://github.com/crkrenn/polis-deploy/tree/feature/docker_compose_localhost_no_math/e2e-tests

We are also talking about the pros and cons of rewriting the javascript code from scratch.

@patcon:

Sorry -- rewrite the server/client code, or your test code?

[patcon]

Once we merge docker-dev, could prob pretty easily start running those puppeteer tests using a GitHub Action like this: https://github.com/ianwalter/puppeteer

Would be great to run on all pushes/prs, but might need to be clever, depending on how often docker builds fail, and how long it takes.

Maybe helpful: https://github.community/t5/GitHub-Actions/Trigger-action-on-schedule-only-if-there-are-changes-to-the/td-p/50569

[patcon]

@ballPointPenguin you asked to claim this todo on the call (#147), so is it cool that I assigned it to you? thanks!

[colinmegill]

+1 for puppeteer

On Sat, May 2, 2020 at 10:44 PM Patrick Connolly notifications@github.com wrote:

@ballPointPenguin https://github.com/ballPointPenguin you asked to claim this todo on the call (#147 https://github.com/pol-is/polis-issues/issues/147), so is it cool that I assigned it?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pol-is/polis-issues/issues/142#issuecomment-623045395, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANQGGIM2QYGZRFRQ4EBTRTRPTLBRANCNFSM4MVAEBTQ .

[ballPointPenguin]

Taking an initial stab at Cypress here https://github.com/pol-is/polis-cypress/tree/master At first I just have create new user and user login running successfully against preprod

[ballPointPenguin]

I could enumerate a pretty long TODO list for e2e testing needs. Maybe that is best done in the cypress repo itslef.

[patcon]

Oh man, really eager to try out the tests! Super stoked for testing

Is the separate repo an temporary place, with the intention to bring into repo after +1's? I'm personally more familiar with tests living inside codebase

[ballPointPenguin]

My general preference, having done it both ways, is that developer tests live in the same repo (unit tests, etc). But an end-to-end suite is often kept outside for a few reasons. This is a kind of opaque or "black box" testing which has no access to or knowledge of the inner workings of the application. It is also easily developed by other contributors and can be merged, upgraded, deployed in a more nimble fashion being divorced from the merge/version/deploy considerations of the application. There are good arguments to be made on both sides, however.

[patcon]

k, sounds good to me :) i suppose form field IDs and routes are the main "interface", and they should be simple to keep in sync

[ballPointPenguin]

and html data-* attributes

[patcon]

Obviously the tests failed (since it's against new UI in preprod), but I took this for a spin to test against a pending PR, and it's pretty wonderful 🎉

I can tell that it's going to feel really nice to validate lack of regressions in PRs quicky by just running:

cd ~/repos/polisServer
docker-compose down && docker-compose build --parallel --no-cache && docker-compose up --detach

cd ~/repos/polis-cypress
CYPRESS_BASE_URL=http://104.248.14.69.xip.io ./node_modules/.bin/cypress run

[patcon]

Once we merge #282, thinking this might be neat to try: https://github.com/cypress-io/github-action

Docs make me think we can run tests headlessly right within github actions, without deploying to some server of our own 🚀

[patcon]

Found out how to run a subset:

CYPRESS_BASE_URL=https://preprod.pol.is ./node_modules/.bin/cypress run --spec 'cypress/integration/client-admin/**/*'

[patcon]

I feel like we're well on our way into this. Closing!