Accessibility testing

[rowlando]

We all know that tooling only gets us so far when it comes to ensuring quality, usability, accessibility... all the 'bilities. We want tools to increase the time spent on hard to automate, exploratory testing that only humans can do well.

For accessibility testing, the industry has settled on axe. Would be good to hear if there are alternatives.

Where there does seem to be differences of opinion is around our use of tools and frameworks that are underpinned by axe.

Would be good to find out what tools we have in use at the moment, no matter whether they are tied to a backend ecosystem, e.g. Ruby, Rails and Cucumber, or are agnostic to backend language and framework? Then we can go on to discuss recommendations, limited use and do not use categories.

@RoseSAK @MalcolmVonMoJ @mayowaadeniyi @naseberry

[naseberry]

I have used a few tools to automate accessibility testing: a11y, AXE and Google Lighthouse (this now uses AXE at its' core).

All the above tools are language agnostic and works well in any CI/CD process. All the above are light on resources, fast and well documented.

The industry may have settled on AXE, however many company use a11y as it is a community project, i.e. bbc a11y Plus it is always good to validate against multiple tools, if possible, as one tools may identify issues that the other may not.

At present we are using an AXE base tool to automate our testing within our RoR applications.

GDS had compared a few automated accessibility checkers - https://alphagov.github.io/accessibility-tool-audit/

What tools have you used? What tools are you currently using?

[MalcolmVonMoJ]

On Apply, we used Cucumber to render the pages and Webhint to scan the pages for accessibility issues. Webhint uses aXe as its main checker, but it also has a whole suite of things that aren't related to accessibility. It caught a security issue in one release.

An ordinary aXe test will fail several elements of the GDS design system, but as these have all been extensively tested and left in the design system's elements, we want these to not trigger a failure. Webhint allows for these sorts of configurations.

On the CLA/FALA service, we are just starting to use Webhint. This decision was made in the knowledge that Apply have stopped using Webhint, at least for the now, and that we cannot use Cucumber to render the pages. Instead we intend to use Cypress to render the pages. Initial tests are positive, and it looks like we are going to go down the Webhint/Cypress route.

Mayowa Adeniyi is leading on this and may have more specific information.

My understanding is that the configuration of Webhint in the Apply service caused some issues which resulted in them suspending it. I understand that this is a result of it being badly configured rather than it being unsuitable.

[RoseSAK]

I joined Apply fairly recently so I may have missed previous conversations about this. Apply had Webhint installed but it had been unusable for some time, for reasons I'm not completely sure of but seemed to be to do with switching from npm to yarn, and not updating/reconfiguring Webhint. We decided to look into alternatives to Webhint before trying to fix it. Peter Hunt suggested the axe-core-gems which is a set of Ruby gems based on aXe, and since we already have cucumber tests we decided to go with axe-core-cucumber. There are also gems to add accessibility testing via other testing frameworks/tools, e.g. rspec (https://github.com/dequelabs/axe-core-gems).

The reasons for deciding on axe-cucumber over Webhint were:

• Webhint was apparently very slow to run the tests, since it has to render the page first (we generally run the cucumber tests headlessly, and our whole test suite still takes well over ten minutes to run).
• Webhint also required a lot of extra configuration, and came with a lot of extra functionality and we thought that for now at least we only really require the accessibility testing part.
• I think it was also a hassle to integrate the Webhint tests into our CircleCI pipeline (again, more config needed).

Axe-cucumber on the other hand runs very quickly, was easy to install and there is minimal setup/configuration. It works in our pipeline with no extra configuration since it's just an extra step in the existing cucumber tests. The steps can also be modified easily to skip certain aspects of the GDS design system which would otherwise fail. It updates automatically as it's just a gem. Axe cucumber also (obviously) uses aXe as the accessibility checker so in theory it should find the same issues as Webhint does.