USWDS-Site: Investigate replacing pa11y with axe-core

[amyleadem]

pa11y-ci 2.x uses an outdated version of pa11y, but upgrading to pa11y-ci 3.x introduces a host of false positives (See notes below for more details).

We should investigate what it would take to use axe-core in in place of pa11y-ci.

[amyleadem]

pa11y 3 notes

Related pa11y and pa11y-ci issues and PRs

• pa11y issue about ignoring rules in axe: https://github.com/pa11y/pa11y-ci/issues/156
• pa11y PR adding ability to ignore axe rules: https://github.com/pa11y/pa11y/pull/592
• pa11y reports false positives for color contrast: https://github.com/pa11y/pa11y/issues/633
• pa11y reports an axe warning as an error: https://github.com/pa11y/pa11y/issues/623

Axe rule set: https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md

Setup - pa11y 2.4.2

• pa11y-ci v2.4.2 (uses pa11y ~5.3.1)

.pa11yci config:

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "concurrency": 2,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude '/*.pdf|next/'

Running Pa11y on 169 URLs:
 > http://localhost:4000/components/accordion/ - 0 errors
 > http://localhost:4000/components/alert/ - 0 errors
 > http://localhost:4000/components/banner/ - 0 errors
 > http://localhost:4000/components/breadcrumb/ - 0 errors
 > http://localhost:4000/components/button-group/ - 0 errors
 > http://localhost:4000/components/button/ - 0 errors
 > http://localhost:4000/components/card/ - 0 errors
 > http://localhost:4000/components/character-count/ - 0 errors
 > http://localhost:4000/components/checkbox/ - 0 errors
 > http://localhost:4000/components/collection/ - 0 errors
 > http://localhost:4000/components/combo-box/ - 0 errors
 > http://localhost:4000/components/data-visualizations/ - 0 errors

Setup - pa11y 3.0.1

• pa11y-ci v3.0.1 (uses pa11y ~6.1.0)

.pa11yci config (Updated concurrency config to use default value of 1 - this prevents failed to run issues):

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}

❯ npm run pa11y-ci:sitemap                                  

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude '/*.pdf|next/'

Running Pa11y on 169 URLs:
 > http://localhost:4000/components/accordion/ - 17 errors
 > http://localhost:4000/components/alert/ - 6 errors
 > http://localhost:4000/components/banner/ - 2 errors
 > http://localhost:4000/components/breadcrumb/ - 2 errors
 > http://localhost:4000/components/button-group/ - 4 errors
 > http://localhost:4000/components/button/ - 2 errors
 > http://localhost:4000/components/card/ - 2 errors
 > http://localhost:4000/components/character-count/ - 2 errors
 > http://localhost:4000/components/checkbox/ - 3 errors
 > http://localhost:4000/components/collection/ - 2 errors
 > http://localhost:4000/components/combo-box/ - 2 errors
 > http://localhost:4000/components/data-visualizations/ - 35 errors

Error details:

❯ pa11y-ci http://localhost:4000/components/alert/
Running Pa11y on 1 URLs:
 > http://localhost:4000/components/alert/ - 6 errors

Errors in http://localhost:4000/components/alert/:

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview)

   <h2 id="alert-preview">Component preview</h2>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(2) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(3) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(4) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(5) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-code)

   <h2 id="alert-code">Component code</h2>

✘ 0/1 URLs passed

Setup - pa11y 3.0.1 with HTML Codesniffer runner

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA"
  }
}

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude '/*.pdf|next/'

Running Pa11y on 169 URLs:
 > http://localhost:4000/components/accordion/ - 0 errors
 > http://localhost:4000/components/alert/ - 0 errors
 > http://localhost:4000/components/banner/ - 0 errors
 > http://localhost:4000/components/breadcrumb/ - 0 errors
 > http://localhost:4000/components/button-group/ - 0 errors
 > http://localhost:4000/components/button/ - 0 errors
 > http://localhost:4000/components/card/ - 0 errors
 > http://localhost:4000/components/character-count/ - 2 errors
 > http://localhost:4000/components/checkbox/ - 1 errors
 > http://localhost:4000/components/collection/ - 0 errors
 > http://localhost:4000/components/combo-box/ - 0 errors
 > http://localhost:4000/components/data-visualizations/ - 0 errors
 > http://localhost:4000/components/date-picker/ - 0 errors

Setup - pa11y 3.0.1 - ignore color contrast

• pa11y v6.1.0
• pa11y-ci v3.0.1 (uses pa11y ~6.1.0)

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}

Passing --ignore color-contrast flag resolves errors

• Running pa11y-ci results in 17 errors: pa11y-ci http://localhost:4000/components/accordion/
• Running pa11y, passing the --runner axe flag results in 17 errors: pa11y --runner axe http://localhost:4000/components/accordion/
• Running pa11y, passing the --ignore color-contrast flag results in 0 errors: pa11y --runner axe --ignore color-contrast http://localhost:4000/components/accordion/

Adding ignore configuration to .pa11yci

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"],
    "ignore": ["error", "warning", "notice"]
  }
}

Running pa11y-ci results in 0 errors: pa11y-ci http://localhost:4000/components/accordion/

Editing ignore configuration to ignore color contrast

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"],
    "ignore": ["color-contrast"]
  }
}