Bringing results to developers

[mgifford]

Is there a way to provide a pass/fail result rather than the whole JSON issue.

I'd like to be able to just crawl my site and not have to hit all of the URLs in the string: axe --dir --tags wcag2a ./axe-results/ openconcept.ca, openconcept.ca/blog, openconcept.ca, https://openconcept.ca/contact

But I would like to know what specific pages are problematic. I don't want to have to crawl through the JSON file to find the needle in the haystack. I like how Lighthouse gives you a number that you can refer to and pass/fail on.

Maybe that isn't possible with axe given the difference between the projects. I'm not sure how Google generates that number for instance.

I mostly am looking for a way to tie this into a CI system so that developers are pushed to go back and figure out what they may have added that is breaking the build. I feel like Axe is close on this, but haven't seen a mechanism to crawl & give that pass/fail that many developers are looking for.

[stephenmathieson]

Something like this should solve the problem you're seeing:

axe-tests.sh

#!/bin/bash

for url in openconcept.ca openconcept.ca/blog openconcept.ca openconcept.ca/contact; do
  if ! ./axe-cli --exit $url > /dev/null; then
    echo "Accessibility issues detected in $url"
    exit 1
  fi
done

We use the --exit flag to signal to axe-cli we want to exit with a non-zero code if errors are found. Additionally we send the axe-cli output to /dev/null in to not output everything to STDIO.

[stephenmathieson]

With the addition of the --stdout flag as proposed in #83, you should be able to opt-into writing results to STDOUT and processing them yourself with jq (or any other command-line JSON processing tool).

[mgifford]

That didn't work, but this almost did: `#!/bin/bash

for url in openconcept.ca openconcept.ca/blog openconcept.ca openconcept.ca/contact; do if ! axe --exit $url > /dev/null; then echo "Accessibility issues detected in $url" exit 1 fi done`

But I'm still getting an error when running:

$ ./axe-tests.sh --stdout axe-webdriverjs deprecated Error must be handled as the first argument of axe.analyze. See: #83 ../../usr/local/lib/node_modules/axe-cli/lib/axe-test-urls.js:85:8 Accessibility issues detected in openconcept.ca

Not sure what I am doing wrong.

[stephenmathieson]

If you update axe-cli and axe-webdriverjs, that error will go away. If you follow the error, you'll see that some old axe-webdriverjs behavior was depreciated.

[mgifford]

I'm still getting the same error:

Mikes-MacBook-Pro-2:~ mikegifford$ npm-check --update ❤️ Your modules look amazing. Keep up the great work. ❤️ Mikes-MacBook-Pro-2:~ mikegifford$ npm update axe-cli Mikes-MacBook-Pro-2:~ mikegifford$ npm update axe-webdriverjs Mikes-MacBook-Pro-2:~ mikegifford$ ./axe-tests.sh --stdout axe-webdriverjs deprecated Error must be handled as the first argument of axe.analyze. See: #83 ../../usr/local/lib/node_modules/axe-cli/lib/axe-test-urls.js:85:8 Accessibility issues detected in openconcept.ca

[stephenmathieson]

Ah, you're right, we're not auto-publishing this package on every commit. We'll need bf68ca77428754f68f5508056acfa03cba015d31 to be published before the deprecation warning goes away on its own.

FWIW you can safely ignore the deprecation warning for now, or turn it off following the instructions provided in the depd package.

[mgifford]

I'm not sure what I'm getting wrong here, but I still can't get it to do a report on more than one URL.

[stephenmathieson]

The program is exiting if it finds accessibility issues. Remove exit 1 from the above bash script if you don't want it to exit.

[mgifford]

Right... that makes total sense..

Now to improve the script you might actually want to actually echo the number of detected errors and pull out the string $error string "5 Accessibility issues detected" and then spit out: echo $error . " in $url"

But that's beyond what I can awk at the moment.

[straker]

Closing as it seems the issue was resolved. Please reopen if that is not the case.