QA test scripts & automation

[sueannma]

Aha! Link: https://diagramlabs.aha.io/features/B-363

[pgesek]

We should also consider doing automated accessibility checks during the Selenium builds.

[pgesek]

@johnhbenetech I've create a proof of concept test suite for MathShare using Maven, Selenium, Cucumber and Axe for accessibility checks. The repository is here:

https://github.com/pgesek/MathShare-tests

I've based it on an existing example I cleaned up a bit. Here is a very simple feature that tests the MathShare interface:

https://github.com/pgesek/MathShare-tests/blob/master/src/test/resources/features/solve_problem.feature

Notice the step: 'Check Accessibility: max 3 violations'. This will run axe-core checks on current site and print the output. If there are more than 3 violations the step fails (3 is the current number of violations on the site).

Let me know if this is fine for an automation and what you would change improve. If we decide this fine I can transfer this repository to Benetech.

[pgesek]

For the record, the current violation output is:

Found 3 accessibility violations: 1) Elements must have sufficient color contrast: https://dequeuniversity.com/rules/axe/2.6/color-contrast?application=axeAPI a) ["#ProblemTitle"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 14.4pt, font weight: normal). Expected contrast ratio of 4.5:1

b) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(1)"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

c) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(2)"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

d) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(3)"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

e) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(4) > .vlist.mfrac > span:nth-child(3) > .uncramped > span"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

f) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(5)"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

g) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(6)"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

h) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(7)"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

i) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(8) > .vlist.mfrac > span:nth-child(3) > .uncramped > span"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

j) ["#ProblemMath > .staticMath.problemTitle > span:nth-child(2) > span:nth-child(2) > .MLmathlive > .MLbase > span:nth-child(9)"] Fix any of the following: Element has insufficient color contrast of 3.03 (foreground color: #ffffff, background color: #5997e2, font size: 17.3pt, font weight: normal). Expected contrast ratio of 4.5:1

k) ["#addStep"] Fix any of the following: Element has insufficient color contrast of 3.97 (foreground color: #ffffff, background color: #007bff, font size: 12.0pt, font weight: normal). Expected contrast ratio of 4.5:1

2) id attribute value must be unique: https://dequeuniversity.com/rules/axe/2.6/duplicate-id?application=axeAPI a) [".mainWrapper > div.header"] Fix any of the following: Document has multiple elements with the same id attribute: topNavigation

3) Form elements must have labels: https://dequeuniversity.com/rules/axe/2.6/label?application=axeAPI a) ["#mathAnnotation"] Fix any of the following: aria-label attribute does not exist or is empty aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty or not visible Form element does not have an implicit (wrapped) Form element does not have an explicit Element has no title attribute or the title attribute is empty

[johnhbenetech]

@pgesek I see support for chrome/firefox, but how much of this might be extensible to Safari/Edge?

My assumption is that Safari is probably a big leap because the selenium driver would need to be running on a mac for it to work? Let me know if I am misunderstanding.

I did some brief research and see some commercial solutions allowing you to run selenium tests in the cloud over 1000+ browser/device combinations. I can look at a few of those to see if any of them are affordable. We can always run one off manual checks on Safari in preparation for big releases.

In the mean time I'll check with our systems team about how they want to put this into jenkins, but it usually takes some time.

[pgesek]

@johnhbenetech I believe there is support for Edge driver in the code, adding Safari is also not an issue code-wise.

However, as you mention, the tests must run on an environment with the given browser installed. If we also want to run our tests on Safari/Edge, we will need to run them on an environment with the given browser installed. I am also aware of the solutions you mention, most notably BrowserStack and SauceLabs. Hooking those up will be easy if we decide to go that way.

[NSoiffer]

The last two warnings should be easy to fix (and should be fixed).

The duplicate id problem should be obvious

The label problem should be fixed by adding the attr aria-labelledby="mathAnnotationHeader" to the textarea element.

[afijal]

@NSoiffer Sure, those will be fixed in #52

[pgesek]

@dserkowski please extend the build to be able to take props like the gecko engine location from the command line, i.e.

mvn clean test -Dwebdriver.gecko.driver=/my/path/gecko

Most likely this will require to define the suefire plugin configuration, but I am not sure. Please also test this with ChromeDriver (add a prop to make it work as well if need be).

[dserkowski]

Ok, I have checked it and updated README. Chrome and Gecko (for Firefox) drivers work well. I don't have the possibility to test it for Safari and Edge.

To run tests you need to execute the following commands:

# for Firefox
mvn clean test -Dbrowser=firefox -Dwebdriver.gecko.driver=/home/user/geckodriver
# for Chrome
mvn clean test -Dbrowser=chrome -Dwebdriver.chrome.driver=/home/user/chromedriver
# for Safari
mvn clean test -Dbrowser=safari -Dwebdriver.safari.driver=/home/user/safaridriver
# for Edge
mvn clean test -Dbrowser=edge -Dwebdriver.edge.driver=/home/user/MicrosoftWebDriver.exe