Functional testing guidelines

cjcenizal commented 7 years ago

Based off https://github.com/elastic/kibana/issues/9106

Let's create guidelines for how to write good functional tests, style_guides/functional_test_style_guide.md.

Content

Define "functional testing" (aka end-to-end testing).
The purpose of functional tests.
How to use Page Objects.
Using data-test-subj (https://github.com/elastic/kibana/pull/8854#discussion_r85557389).
When to take screenshots.
Links to relevant resources, e.g. page objects.
Cross-link to public docs https://www.elastic.co/guide/en/kibana/master/development-plugin-functional-tests.html

ycombinator commented 7 years ago

I'm starting to instrument some code with data-test-subj attributes. Looking around the codebase for existing usages of data-test-subj, specifically any conventions around the values of this attribute, I found a mix of camelCase and lisp-case being used.

It looked like most of the values used camelCase, so I'd propose that we make that the convention when this style guide is created.

cjcenizal commented 7 years ago

@ycombinator Great call. I think the only appropriate use case for hyphens would be if you need to dynamically generate the selector and you don't want to do string manipulation:

<div ng-repeat="thing in things" data-test-subj="thingItem-{{:thing.name}}">

cjcenizal commented 7 years ago

CC @tylersmalley

cjcenizal commented 6 years ago

CC @chrisronline

cjcenizal commented 6 years ago

Confirm success/failure the same way a user would

For example, use the presence of a toast (https://github.com/elastic/kibana/pull/15749/files#diff-900adf3e48329f3b99bb703081058de3R327) or a callout (https://github.com/elastic/kibana/pull/21250) to verify that the outcome you expected has occurred, whether that outcome is success, failure, or something else. If a toast or callout doesn't exist, then consider adding one.

Verify location the same way a user would

For example, check the last breadcrumb (https://github.com/elastic/kibana/pull/15749) or the page title (https://github.com/elastic/kibana/pull/21245) to verify that you've successfully navigated to a location and the UI is ready to be used.

Don't assert against complete text content

Checking for full equality with an entire sentence or paragraph is paranoid and brittle. Find the part of that sentence of paragraph which is the essence of that paragraph, e.g. an error code or a noun like "5 visualizations were successfully deleted", and check that the string includes the important substring. This will be more likely to withstand future changes without breaking the test.

cjcenizal commented 6 years ago

Here's an earlier PR for adding a functional tests README, which contains some useful information on the TestSubjects and Find services: https://github.com/elastic/kibana/pull/21175

cjcenizal commented 6 years ago

More insight on the best way to interact with the UI here: https://github.com/elastic/kibana/pull/21123#issuecomment-407486888

cjcenizal commented 6 years ago

Wait until elements have finished animating before interacting with them

This is borrowed from https://medium.freecodecamp.org/how-to-write-reliable-browser-tests-using-selenium-and-node-js-c3fdafdca2a9. In https://github.com/elastic/kibana/pull/21302 I kept running into problems where I would try to dismiss a toast by clicking on it, only for the click to silently fail. This was because the toast was still moving when the click was sent, so the click ended up missing. I solved this by waiting for the toast's opacity to be 1 before trying to click it.

cjcenizal commented 6 years ago

Don't use findDisplayed* methods

These methods look only for visible objects, which is slower and prone to flakiness. The should really only be used in edge cases where the visibility of an element can’t be ensured in advance. Instead of using findDisplayed* we should just use find*.

elastic / kibana