USWDS-Site - CI: Improve pa11y testing

[mejiaj]

Pa11y tests are slow, crash often, and test every page.

Tests take 7min on average and fail often. Development for the tool lags behind their other, main, product see Pa11y-CI [GitHub] releases. Last release was Nov 2023.

If we want to continue using this tool, we should:

• [ ] Identify which URLs we want to test.
• [ ] Consider ignoring some sections (like blog).
• [ ] Consider setting a task that runs at a lower frequency (weekly/monthly).
• [ ] Test another CI tool (like GitHub actions).

Additional information

Locally it takes a while too. From desktop test - npm run pa11y-ci:sitemap:

Errors in http://localhost:4000/documentation/settings/:

 • Error: Pa11y timed out (120000ms)

✘ 174/175 URLs passed

To do

• [ ] Remove pa11y from circleCI

  • 2689

• [ ] Explore adding a pre-commit hook that runs pa11y
• [ ] Explore if pa11y-ci performs better with GitHub actions
• [ ] Explore if we can run pa11y only on changed pages (not the whole site)
• [ ] Explore if we can upgrade to pa11y-ci v3

  • 1589

• [ ] Explore running axe-core locally instead

  • 1884

• [ ] Explore running pa11y locally instead

[aduth]

If it's useful for reference, we had migrated off pa11y to HTML_CodeSniffer in our project in https://github.com/18F/identity-design-system/pull/285 , though (a) we already had an integrated browser driver to fall back on and (b) we were only using the simpler HTML_CodeSniffer (default) tests, not Axe (unsure what's used in this project).

[sknep]

For cloud.gov Pages, we're looking into axe-core and reporting the JSON of results in HTML, similar to https://lpelypenko.github.io/axe-html-reporter/ -- happy to chat with USWDS folks about it if you're game!

[mejiaj]

Thank you both! Last week pa11y had a lot of crashes and yesterday it seemed okay. Overall the reliability is a big concern.

We moved from axe-core to pa11y because of the configurability and the ability to test against a sitemap made it really easy to integrate.

Currently, we're not using a browser driver and we test against WCAG AA.