Improve accuracy and ci speed

[j-mendez]

Using pa11y leaves out up to 10-35% issues with htmlcs and axe-core opposed to using a11y-js. The repo provides test and benchmarks showing the missing issues not being covered along with drastic speed issues.

I kept seeing this repo popping up on my GitHub feed and it’s an eye sore to see pa11y being used with better alternatives at hand.

Swapping the ci action with https://github.com/a11ywatch/github-actions should be almost a drop in replacement with minor changes and will provide the results with better accuracy with the ability to finish in milliseconds to seconds vs minutes to several hours.

[mgifford]

We will check it out. Thanks @j-mendez for bringing it to our attention.

[dmundra]

Hey @j-mendez, trying it out and had some comments and questions for you:

• I think the sitemap is automatically selected. When running against the local site with pa11y we specific the sitemap but also do a search and replace to use local site URLs because Jekyll generates sitemap with the main URL. See https://github.com/CivicActions/accessibility/blob/main/package.json#L13. I could probably add a separate step to replace the URLs in the sitemap but wanted to ask.
• When trying the site against the live site https://github.com/CivicActions/accessibility/actions/runs/5028493792/jobs/9019248296?pr=697 I got some results but it didn't comment on the PR. I am not why that is the case and why the pipeline still passed.
• Do you think we should use https://github.com/a11ywatch/kayle instead or stick with the github action?

Thank you and let me know if should post these questions in https://github.com/a11ywatch/github-actions instead.

[j-mendez]

Hello @dmundra, at the moment the github action does not allow replacing the url, I looks like a worthy add based on the use case above, atm the crawler will only respect links found under the domain of the url passed in. Seems like a reasonable feature just not sure about timeline for adding it. Personally I start the local instance in dev mode and usually all the links are configured dynamically to http://localhost.

In order to receive the github comments use the config var LIST set to true and this will output a pass fail list or set the FAIL_ERROR_COUNT var.

- uses: a11ywatch/github-action@v1.14.0
  with:
    WEBSITE_URL: ${{ secrets.WEBSITE_URL }}
    SITEMAP: true
    FAIL_ERRORS_COUNT: 15
    LIST: true

I would only use kayle fully by itself for small to medium websites. Integrating with kayle is quite flexible with any headless runner or even non headless as a linter.

[j-mendez]

@dmundra If you have any more questions feel free to followup on any of the github repos as well. FYI the github action takes a bit longer to start up for the first time since it is downloading and installing, afterwards it should be a lot quicker so the rewards of the speedy testing coming into play. Thanks for checking out the project as well!