18F / 18f.gsa.gov

The 18F website
https://18f.gsa.gov
Other
289 stars 310 forks source link

Add nightly pa11y scan #3832

Open caleywoods opened 1 month ago

caleywoods commented 1 month ago

Feature suggestion

Create a GitHub/CircleCI action that runs a nightly pa11y scan. If this scan results in pa11y finding issues, it should create a GitHub issue with the details.

caleywoods commented 1 month ago

Research findings/approach:

  1. Create a scheduled pipeline run in CircleCI
  2. The .circleci/config.yml needs to have its "Jekyll build" and "Run pa11yci" steps adjusted. "Jekyll build" no longer needs to export the pa11y_targets file contents to the env var nor does it need to source $BASH_ENV and the "Run pa11yci" step should be: ./serve-pa11yci && npm run pa11y-sitemap to do a full scan
  3. A final step should be added that checks to see if the pa11y scan returned any failures and if so it should run the terminal command curl \ -X POST \ -H "Accept: application/vnd.github.v3+json" \ https://api.github.com/repos/18f/18f.gsa.gov/dispatches \ -d '{"event_type":"pa11y_scan_failure", "client_payload": {"message":"failed_urls_list"}} to dispatch an event to our repo which will trigger the GitHub action we have created that listens for the GitHub repository_dispatch event

The logic to determine if pa11y failed may need to make use of the $BASH_ENV stuff again where we export the failed URL list to an env var in the "Run pa11yci" step and then read it back out to send in the client_payload object when calling the GH API.

caleywoods commented 2 weeks ago

This is working, see #3878 as an example. This issue will be closed soon by a PR.

caleywoods commented 2 weeks ago

@beechnut If you've got the spoons, can you review the rough wiki entry I've created for this? I'm linking directly to the Jekyll plugin portion but the whole thing could use a review. I know we're also in the process of porting the site to Eleventy so if it makes more sense to just let the Jekyll plugin section live as is for now and not spend time on it that makes sense.

https://github.com/18F/18f.gsa.gov/wiki/CircleCI-scheduled-nightly-pa11y-scan#jekyll-plugin