search

mozilla-services / push-dev-dashboard

Developer dashboard for the Mozilla web push service
https://dev-dashboard.deis.dev.mozaws.net/
Mozilla Public License 2.0
7 stars 6 forks source link

Set up QA test suite for stage & prod deployments #200

Open groovecoder opened 8 years ago

groovecoder commented 8 years ago

Maybe a single Selenium end-to-end test:

In order to test the entire dashboard, the Selenium test will need access to a fixture push application. Options are:

groovecoder commented 8 years ago

@davehunt - what do you think about the above? Do we have other selenium tests that use or span multiple sites/pages?

davehunt commented 8 years ago

It's not common, and chaining multiple sites in an e2e test increases fragility as you'll be at the mercy of several services that you may not have control over. UI integration tests are already rather brittle due to latency and their dependency on the underlying DOM of the application under test.

I can't say I fully understand the context of this issue, but I would generally recommend only writing UI automation for sites you have full control over, and if you do have to interact with third party sites keep this to a minimum. Where possible use documented APIs instead.

For signing in via Firefox Accounts, we do actually maintain a page object model as this is used in a number of our automation suites.

groovecoder commented 8 years ago

@davehunt in that case, we may need to do a manual QA check on the stage & prod servers after each deploy. Is that alright with you - do we have someone in QA who can commit to that?

groovecoder commented 8 years ago

I'm going to go ahead and try to make a selenium test suite that uses https://mozilla-services.github.io/WebPushDataTestPage/ to end-to-end test the dashboard.

groovecoder commented 8 years ago

@davehunt, @pdehaan - I've got selenium tests started in the repo. How can I start making them run on our QA server? I assume we have a Jenkins server with Selenium that I can use?

rpappalax commented 8 years ago

@davehunt @groovecoder I agree with Dave that tests that are tied to a remote service can be brittle and aren't always ideal. For verifying service deployments, however, the reason for being of many of our tests is to verify that the actual service is running properly (i.e. when getting staged). If you know that your tests will be "flaky", then just be aware that you may have the occasional failure. The rest would be up to you to know when a test is "too" flaky due to latency issues or tolerably flaky giving you what you want. But getting back to Dave's point - if you don't need to tie your test to a remote service, that's always better.

groovecoder commented 8 years ago

Note to self: Start a push-dev-dashboard/ directory in services-test repo with e2e-test/run like https://github.com/mozilla-services/services-test/blob/dev/demo/e2e-test/run

groovecoder commented 8 years ago

Adding push-dashboard to services-test here: https://github.com/mozilla-services/services-test/pull/88

groovecoder commented 8 years ago

@rpappalax I just updated this branch with an expanded test that gets all the way thru:

  1. Signing into the dashboard with an existing FxA
  2. Creating a new push app with a VAPID key
  3. Validating the VAPID key by signing a unique "aud" claim payload JWT

The last bit we could do would be to actually send a push message with the VAPID headers and then check that it shows up on the dashboard. But that tests way more than just the dashboard ... i.e., it tests:

In addition, it means that the test will need to configure the Firefox browser used to use the dev/stage/prod autopush service.

Let's plan to get into that next week though ... there may be some parts of it we decide to skip or fake.

groovecoder commented 8 years ago

@rpappalax - the test_e2e module is now merged to master. I'm leaving this open because it needs https://bugzilla.mozilla.org/show_bug.cgi?id=1273260 to work automatically.