dequelabs / axe-cli

[Deprecated] A command-line interface for the aXe accessibility testing engine
Mozilla Public License 2.0
430 stars 35 forks source link

Questions about the exit code #57

Closed rianrietveld closed 6 years ago

rianrietveld commented 6 years ago

I understand that using -q I can force axe-cli to generate an exit code which would allow me to fail a CI build based on the results of a scan.

Now there are a couple of questions I have about this.

  1. How is the exit code determined. Will any notice generate the 1 exit code? Or is it only violations ? What effect do "to review" notices have on the exit code ? Is it possible to, for instance, force both violations + to review to exit with 1? Is it possible to configure axe to only exit with 1 on violations ?
  2. Is it possible to "downgrade" certain violations to "to review" status based on a configuration in the json file ? Or for that matter, to up the error level of certain "to review" items to full violations?

In other words, I'm looking for more detailed information about how to fine-tune the tests being run.

If the above is not (yet) possible, could this please be regarded as a feature request.

/cc @jrfnl

WilcoFiers commented 6 years ago

-q or --exit will result in an exitcode 1 if there are any violations. Not for review items (results.incomplete). There is currently no way to do that. Nor is there a standard way to modify the default ruleset. What you could do is amend the axe.js file with your custom configuration. So you'd add an axe.configure(...) call at the end of the axe.js file, which contains any modifications you'd want to see.

At the point where you start modifying and developing custom configurations for axe-core, I would suggest you start looking at using WorldSpace Attest instead. It is designed to help you design, distribute, and manage custom configurations and custom rules.

rianrietveld commented 6 years ago

@WilcoFiers thanks for the info :-)