[WIP] Add accessibility scan

18F / concourse-compliance-testing

Concourse CI assets for Compliance Toolkit

https://compliance-viewer.18f.gov/

Other

17 stars 7 forks source link

[WIP] Add accessibility scan #117

Open gemfarmer opened 7 years ago

gemfarmer commented 7 years ago

⚠️ WIP ⚠️

Closes #109

This PR will add a job, accessibility-scan, that uses pa11y-crawl to check 508 compliance on target sites.

Done when:

[ ] accessibility-scan crawls hard-coded URL
[ ] accessibility-scan crawls project-specific URLs
[ ] accessibility-scan crawls project-specific URLs only if that target is configured to do so
[ ] accessibility-scan is triggered nightly
[ ] pa11y-crawl outputted resources.json is bundled as a slack notification

cc @mugizico @afeld

gemfarmer commented 7 years ago

Unfortunately, I found a related bug that will snare our attempts to crawl Federalist domains.

gemfarmer commented 7 years ago

@afeld, @mugizico and I wrote a pipeline config that passes. We have now added it to the bottom of the pipeline.yml file, but are unable to test it locally, and don't have permissions to the cloud.gov concourse instance.

So, I was wondering if you could test it for us, or let us know how we can test that configuration locally. 😄 Thanks!

gemfarmer commented 7 years ago

@mugizico @afeld UPDATE: I spoke with @stvnrlly about some of the issues with the existing pa11y-crawl tool and he pointed me to some of the improvements that have recently been made to pally-ci that allow for crawling.

In its current form, pa11y-ci allows for site crawling via a sitemap.xml file or via a .pa11yci config file that specifies a JSON array of URLs to check. The tool is in its infancy, but their roadmap is pretty ambitious, and they have the fully weight of pa11y behind them.

In the short-term, for this particular issue, I propose that we create a tool that conditionally 1) crawls the site and creates a pa11yci JSON config file to use 2) uses the sitemap.xml for the site.

We should also probably promote all 18F sites having a sitemap.xml

gemfarmer commented 7 years ago

Newer version, using pa11y-ci:

jobs:
  - name: accessibility-scan
    plan:
    - task: run-pa11y
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: node
        run:
          path: sh
          args:
          - -exc
          - |
            npm install -g pa11y-ci
            npm install -g phantomjs
            pa11y-ci --json --sitemap https://micropurchase.18f.gov/sitemap.xml

gemfarmer commented 7 years ago

@mugizico

TODO:

[x] add back in to pipeline.yml
[x] I think we will need to output the JSON to a file that has the results
[x] text_file: accessibility-scan/summary.txt

gemfarmer commented 7 years ago

NOTE: @mugizico and I made pa11y-ci work locally and as a CI tool for 18F site. You can see that work here: https://github.com/18F/18f.gsa.gov/pull/1924