search

alphagov / govuk-frontend

GOV.UK Frontend contains the code you need to start building a user interface for government platforms and services.
https://frontend.design-system.service.gov.uk/
MIT License
1.17k stars 321 forks source link

Enhance automated accessibility testing in our development pipeline #3041

Open dav-idc opened 1 year ago

dav-idc commented 1 year ago

What

Timeline: Ongoing work, with a strong focus until March 2023, and then continuing afterwards Priority level: 3 – what reduces risk of inaccessible implementations Category: Design system fundamentals

Potential tasks

Epic lead

Not yet assigned

Why

As of November 2022, there are limitations to how the team implements jest-axe as part of our automated testing:

  1. The team only tests the first example code snippet for each component and pattern – most of our components and patterns have multiple example code snippets.
  2. The team only tests the static HTML version of example code snippets, as found in the HTML tab – some components and patterns include JavaScript that modifies the HTML, which tests do not capture.

Who needs to work on this

Developers, accessibility specialist

Who needs to review this

Developers, tech leads

Initial goals

Future goals

dav-idc commented 1 year ago

Note: this is a duplicate of an older placeholder epic. I've closed the previous issue: https://github.com/alphagov/govuk-design-system/issues/1937

dav-idc commented 1 year ago

Note: this replaces an other older issue related to tooling: https://github.com/alphagov/govuk-frontend/issues/1971

NickColley commented 1 year ago

Just so you know one limitation of jest-axe is that it runs using JSDOM so checks like colour contrast checks do not work. Moving towards something where a real browser is orchestrated (which would also allow testing between interaction states) would improve the coverage of axe-core testing.

colinrotherham commented 1 year ago

@davidc-gds @NickColley Have either of you tried @axe-core/puppeteer (from Deque Systems)?

dav-idc commented 1 year ago

Hey @colinrotherham I've only heard of Puppeteer, and I'm not sure how it compares to jest-axe (or even if comparing the two would be a valid apples-to-apples comparison).

That being said, I can think of a few reasons to warrant further investigation into Puppeteer:

colinrotherham commented 1 year ago

Thanks @davidc-gds it definitely looks handy

But yeah it wouldn't be like-for-like as jest-axe adds nice "reporting" for us

Just so you know one limitation of jest-axe is that it runs using JSDOM so checks like colour contrast checks do not work. Moving towards something where a real browser is orchestrated (which would also allow testing between interaction states) would improve the coverage of axe-core testing.

@NickColley I'd imagine jest-axe could use Puppeteer's Page.setContent() to render static HTML just like JSDOM? Once rendered, could pass the page object into @axe-core/puppeteer versus passing an HTML element into axe-core?

NickColley commented 1 year ago

I think unless there's a really compelling way to opt into a real browser environment I'd not change jest-axe.

Instead my gut feeling it's better to pick a good orchestration suite e.g. something like https://testcafe.io/ or https://go.cypress.io/ (used by the GOV.UK Prototype Kit project) and run axe as part of that.

This way you can write easier to maintain tests that interact with components in a real browser environment and run axe between the states.

colinrotherham commented 1 year ago

Helped out with @axe-core/puppeteer and shared some of our Puppeteer v19 fixes:

After their next release we'll be free to add real browser colour contrast testing etc 😊

dav-idc commented 1 year ago

@colinrotherham has another PR in the works to cover off pretty much all 3 of the potential tasks listed in this activity.

If the PR is successfully merged, this may represent a successful completion of this activity, at least for its initial scope.

It also looks like it will resolve this longstanding issue related to improving automated accessibility testing:

colinrotherham commented 1 year ago

We merged it @davidc-gds 😊

Would be good to confirm if we have any missing tests such as the Cookie Banner UI states

dav-idc commented 1 year ago

Hi @colinrotherham! Where would we be able to review which tests we currently have? I don't think I'm aware of the background info on what happened with the cookie banner UI states you've mentioned.

colinrotherham commented 1 year ago

I've set it up to grab all the examples (nearly 600) from the GOV.UK Frontend Review app

For example these listing pages:

Bear in mind that some components have "hidden examples" which you can navigate to:

You won't find these linked in the Review app, but can see them in each component's config: src/govuk/components/button/button.yaml#L152

They're typically for edge cases used by tests

  • Research the feasibility of testing example code snippets with dynamic JavaScript and multiple states, such as the cookie banner and character count components.

I mentioned the Cookie Banner (see examples) as it's not exactly set up how you'd find it on a service and it fit the "multiple states" point you made under Potential tasks

Might be good to identify components like this and add Axe tests to the Full page examples instead?

https://govuk-frontend-review.herokuapp.com/full-page-examples/cookie-banner-client-side https://govuk-frontend-review.herokuapp.com/full-page-examples/cookie-banner-essential-cookies https://govuk-frontend-review.herokuapp.com/full-page-examples/cookie-banner-server-side

dav-idc commented 1 year ago

After chatting with @colinrotherham, it appears that the initial 3 'potential tasks' are now resolved (which is amazing!)

This issue will remain open, because there are future opportunities for further enhancement. I've listed 3 potential future enhancements in the issue description.

mgifford commented 1 year ago

I'm a big fan of this approach. You can see our implementation here with our own accessibility site https://accessibility.civicactions.com/posts/automated-accessibility-testing-leveraging-github-actions-and-pa11y-ci-with-axe

We've also taken some steps forward with the Drupal community too https://www.thedroptimes.com/30928/drupal-takes-step-forward-in-accessibility-with-automated-testing-integration

This is all just part of shifting accessibility left.

dav-idc commented 1 year ago

We're enhancing our development processes by adding the accessibility-alt-text-bot to our GitHub Actions.

Here's the GitHub issue where we add it:

Learn more about the GitHub enhancement.

cmorten commented 9 months ago

Ended up here following a nice gov.uk design system blog post. Noticed the “future” point of “Research on automated testing focussed on screen reader outputs is completed and potential options are pursued” - bit of shameless self-promotion but would be happy to talk through a package I maintain for just that https://www.guidepup.dev, and as a starter for 10 of other options in the wild there is also a list of alternatives at https://github.com/guidepup/guidepup#similar which hopefully might be of use.