search

uswds / uswds-site

USWDS website and documentation
https://designsystem.digital.gov
Other
187 stars 141 forks source link

USWDS-Site: Upgrade to pa11y 3 #1589

Open mejiaj opened 2 years ago

mejiaj commented 2 years ago

Description

Upgrading to pa11y 3 should fix our jekyll crashing issues.

Additional information

Blockers to upgrading

amyleadem commented 1 year ago

Marking this as blocked until pa11y-ci resolves the false positive issue described in https://github.com/pa11y/pa11y/issues/623.

Other notes on upgrading to pa11y-ci 3.0.1 issues can be found in this pull request: https://github.com/uswds/uswds-site/pull/1855

Some highlights:

New errors appeared

After upgrading to pa11y-ci v3.0.1, we encountered hundreds of new errors after running npm run pa11y-ci:sitemap. After investigation, it was determined that many of these errors were false positives related to color-contrast, and several were valid errors that needed to be remediated.

False positives with pa11y-ci

The false positives were associated with items that had a background image, a background gradient, and/or a pseudo element (The most common flagged items were accordion buttons and external links). When running Axe manually using Chrome developer tools, it was discovered that these same items were being reported, but were not flagged as errors. Instead, they were flagged as items that needed review. On manual check, it was determined that these contrast ratios were well within the acceptable range.

Known issues in pa11y-ci

There is a reported issue with pa11y-ci that appears to flag errors that Axe returns as "warning" or "incomplete: https://github.com/pa11y/pa11y/issues/623

amyleadem commented 1 year ago

Opened issue #1884 to investigate using axe-core as an alternative to pa11y-ci.

amyleadem commented 3 months ago

Notes

I copied these notes from this comment in #1884:

pa11y 3 notes

Related pa11y and pa11y-ci issues and PRs

Axe rule set: https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md

Setup - pa11y 2.4.2

.pa11yci config:

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "concurrency": 2,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}
> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude '/*.pdf|next/'

Running Pa11y on 169 URLs:
 > http://localhost:4000/components/accordion/ - 0 errors
 > http://localhost:4000/components/alert/ - 0 errors
 > http://localhost:4000/components/banner/ - 0 errors
 > http://localhost:4000/components/breadcrumb/ - 0 errors
 > http://localhost:4000/components/button-group/ - 0 errors
 > http://localhost:4000/components/button/ - 0 errors
 > http://localhost:4000/components/card/ - 0 errors
 > http://localhost:4000/components/character-count/ - 0 errors
 > http://localhost:4000/components/checkbox/ - 0 errors
 > http://localhost:4000/components/collection/ - 0 errors
 > http://localhost:4000/components/combo-box/ - 0 errors
 > http://localhost:4000/components/data-visualizations/ - 0 errors

Setup - pa11y 3.0.1

.pa11yci config (Updated concurrency config to use default value of 1 - this prevents failed to run issues):

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}
❯ npm run pa11y-ci:sitemap                                  

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude '/*.pdf|next/'

Running Pa11y on 169 URLs:
 > http://localhost:4000/components/accordion/ - 17 errors
 > http://localhost:4000/components/alert/ - 6 errors
 > http://localhost:4000/components/banner/ - 2 errors
 > http://localhost:4000/components/breadcrumb/ - 2 errors
 > http://localhost:4000/components/button-group/ - 4 errors
 > http://localhost:4000/components/button/ - 2 errors
 > http://localhost:4000/components/card/ - 2 errors
 > http://localhost:4000/components/character-count/ - 2 errors
 > http://localhost:4000/components/checkbox/ - 3 errors
 > http://localhost:4000/components/collection/ - 2 errors
 > http://localhost:4000/components/combo-box/ - 2 errors
 > http://localhost:4000/components/data-visualizations/ - 35 errors

Error details:

❯ pa11y-ci http://localhost:4000/components/alert/
Running Pa11y on 1 URLs:
 > http://localhost:4000/components/alert/ - 6 errors

Errors in http://localhost:4000/components/alert/:

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview)

   <h2 id="alert-preview">Component preview</h2>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(2) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(3) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(4) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-preview-content > div:nth-child(5) > div > p > a)

   <a class="usa-link" href="javascript:void(0);">consectetur adipiscing</a>

 • Elements must have sufficient color contrast
   (https://dequeuniversity.com/rules/axe/4.4/color-contrast?application=axeAPI)

   (#alert-code)

   <h2 id="alert-code">Component code</h2>

✘ 0/1 URLs passed

Setup - pa11y 3.0.1 with HTML Codesniffer runner

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA"
  }
}
> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude '/*.pdf|next/'

Running Pa11y on 169 URLs:
 > http://localhost:4000/components/accordion/ - 0 errors
 > http://localhost:4000/components/alert/ - 0 errors
 > http://localhost:4000/components/banner/ - 0 errors
 > http://localhost:4000/components/breadcrumb/ - 0 errors
 > http://localhost:4000/components/button-group/ - 0 errors
 > http://localhost:4000/components/button/ - 0 errors
 > http://localhost:4000/components/card/ - 0 errors
 > http://localhost:4000/components/character-count/ - 2 errors
 > http://localhost:4000/components/checkbox/ - 1 errors
 > http://localhost:4000/components/collection/ - 0 errors
 > http://localhost:4000/components/combo-box/ - 0 errors
 > http://localhost:4000/components/data-visualizations/ - 0 errors
 > http://localhost:4000/components/date-picker/ - 0 errors

Setup - pa11y 3.0.1 - ignore color contrast

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}

Passing --ignore color-contrast flag resolves errors

Adding ignore configuration to .pa11yci

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"],
    "ignore": ["error", "warning", "notice"]
  }
}

Running pa11y-ci results in 0 errors: pa11y-ci http://localhost:4000/components/accordion/

Editing ignore configuration to ignore color contrast

{
  "defaults": {
    "userAgent": "pa11y",
    "timeout": 120000,
    "standard": "WCAG2AA",
    "runners": ["axe"],
    "ignore": ["color-contrast"]
  }
}