How to integrate AxE with a comprehensive gap analysis and testing strategy

[mmikitka]

Hello,

Firstly, thank you for releasing AxE to the public. I have been impressed by the code structure, attention to detail, and the clear values outlined in the manifesto.

I am planning to integrate AxE with a WCAG AA evaluation process for a public institution, and I am trying to associate the WCAG success criteria with a confidence value associated with the AxE tests so that manual testing gaps can be identified (i.e., criteria, or parts of, that require manual testing).

I am trying to relate each AxE rule to the following WCAG dimensions:

• Level (A, AA) - Already covered by AxE rule metadata
• Principle - Not included in AxE rule metadata
• Guideline - Not included in AxE rule metadata
• Success Criteria - Not included in AxE rule metadata
• Technique (sufficient, advisory, failure) - Not included in AxE rule metadata

With such metadata in place, I can better assess what needs to be tested manually and what can be tested with AxE.

Can you provide any guidance on relating these WCAG dimensions to a comprehensive accessibility testing strategy that includes AxE as an automated testing tool?

Additionally, I am willing to assist with any metadata manipulation and programming tasks, but I would appreciate your guidance on the above before proceeding with any code changes.

Thank you.

[dylanb]

Hey @mmikitka thank you for using axe-core. We are pleased you find it useful.

The Level, Principle, Guideline and Success Criteria are all included in the aXe rule metadata. For example the label rule: https://github.com/dequelabs/axe-core/blob/master/lib/rules/label.json#L5

You can see that we think it is:

• Level A, and
• Principles 1 and 3, and
• Guidelines 1.3 and 3.3 respectively, and
• Success Criteria 1.3.1 and 3.3.2 respectively

We also map it to the relevant Section 508 paragraphs...

We do not map to techniques or failures. These should (if they get added) map onto the checks: https://github.com/dequelabs/axe-core/tree/master/lib/checks

For example: the explicit label check https://github.com/dequelabs/axe-core/blob/master/lib/checks/label/explicit.js maps to the sufficient technique H44. If you wanted to help by adding these mappings to the checks. Please submit a pull request. This would undoubtedly be a great addition to the project.

[mmikitka]

Hi @dylanb - Thanks for the quick and thoughtful response.

I notice that the principles, guidelines, and success criteria are present, but I am looking for a metadata structure that is more amenable to robust reporting needs. Something like the following:

1. Move/copy tags into the metadata property
2. Move implicit meaning out of the tag value and into more explicit metadata categories.
3. Specify the completeness of the test with respect to human intervention i.e., in addition to executing this automated test, what other automated tests and/or human interventions are required to sufficiently reach completeness?

Continuing with the label rule, the label.json metadata may look like:

"metadata": {
  "criteria": [{
    "standard": "wcag",
    "level": "A",
    "principle": "1"
    "guideline": "1.3",
    "success": "1.3.1",
    "techniques": {
       "sufficient": ["ARIA16", "H44", "H65"]
    },
  },
  {
    "standard": "wcag",
    "level": "A",
    "principle": "3"
    "guideline": "3.3",
    "success": "3.3.2",
    "techniques": {
       "sufficient": ["ARIA9", "H44", "H65"]
    },
  }]
}

Then, we can have separate configuration, outside of the rules and checks, in which the sufficiency of the automated tool for meeting the required success criteria and any relevant human interventions are specified.

For example:

{
  "criteria": [{
    "standard": "wcag",
    "level": "A",
    "principle": "1"
    "guideline": "1.3",
    "success": "1.3.1",
    "humanTechniqueChecks": [
      ["ARIA16", "Is the label succinct and meaningful?"],
      ["H44", "Is the label succinct and meaningful?"],
      ["H65", "Is the label succinct and meaningful?"],
      ["ARIA20", "Are the regions structured in an intuitive and easy to navigate manner?"],
      ["G141", "Are the headings structured in an intuitive and easy to navigate manner?"],
      ...
    ],
    "humanFailureChecks": [
      ["F34", "Is white space used to create multiple columns in plain text context?"],
      ...
    ]
}

1.3.1 techniques and failure checks sourced from: https://www.w3.org/TR/UNDERSTANDING-WCAG20/content-structure-separation-programmatic.html

And, stepping back for a moment, are you aware of a testing framework that provides a way of navigating the divide between the completeness of automated testing tools and the need for human intervention?

Please advise on how to move forward. I'm available to help with coding and metadata entry.

Thanks for considering.

matt

[dylanb]

@mmikitka the current data could be used to generate your first structure (if we had the techniques captured). So the question I have is: are you looking for something human-readable or machine-readable?

That being said, let me give you the aXe approach: the data in the system is primarily designed to be machine-readable, with the exception of the rule "description" and "help" and the check "messages".

But, you could ignore all the existing human-readable messages and store your own set of messages independently.

This machine-readable approach has been taken primarily to allow aXe to be easily integrated into any system, including one that must be translated into many languages.

So I keep coming back to: the first step would be to add the techniques and failure information to the existing checks.

Your second structure is outside the scope of the aXe library. It is definitely interesting and might fit within the scope of the aXe extensions. but you might be best off, creating a separate project where you encode the information so that it can be used by anyone (including the aXe extension).

Also, take a look at this project - which is related https://github.com/w3c/wcag-act

[mmikitka]

@dylanb thanks for the response. I'll answer your questions inline:

So the question I have is: are you looking for something human-readable or machine-readable?

The metadata should be machine-readable.

So I keep coming back to: the first step would be to add the techniques and failure information to the existing checks.

I agree with you; I was just proposing an alternative metadata scheme, but it sounds like the only modification that you're willing to make is the addition of techniques to the parent "metadata" map.

Your second structure is outside the scope of the aXe library.

I agree with you. I wanted to give context to my use-case.

Also, take a look at this project - which is related https://github.com/w3c/wcag-act

Yes, thank you. I'll be following the project closely.

[dylanb]

@mmikitka we are willing to make any changes that make sense. The metadata are part of the external API, so changing them means impacting everyone who is using them. This means that we would have to make this change in a backwards compatible way and at the very least, deprecate the existing format and support it for some period of time (perhaps up until version 3).

WDYT about implementing your format as a separate reporter in V2 so we can test it in preparation for V3?

[WilcoFiers]

Hi guys. Sorry to jump into this conversation a little late. The kind of data Matt is looking for is available in other projects. I'm not sure it makes sense for us to maintain it inside of axe, as it would simply be a duplicate of work that's happening elsewhere.

Probably the best source of data, if you want techniques is the W3C's quickref itself. It has all the technique data you could possibly want, and you know it'll be up to date as this dataset is actually used by the W3C. https://github.com/w3c/wai-wcag-quickref/blob/gh-pages/_data/wcag2.json

Alternatively, there's also a project I created for the WCAG-EM Report tool: Which has several translations of WCAG, and follows the same format as the quickref one: https://github.com/WilcoFiers/wcag2json/tree/master/wcag2-json

Let me know if some of this work can help @mmikitka. It should be pretty straight forward to write a little reporter that uses one of these data sources.

[mmikitka]

@WilcoFiers Thank you so much -- the JSON data will be very helpful.

Now, I can do the following outside of aXe:

1. Map aXe rules and checks to the Success Criteria and/or the Techniques in the W3C quickref JSON data.
2. Assign automated, semi-automated, manual methodology states to each sufficient criteria; if a methodology state is semi-automated or manual, then include steps for human intervention

@WilcoFiers are you aware of existing test systems/services that provide the ability to distinguish automated checks from manual checks for a given automated accessibility testing tool, and prompt the user for human intervention wherever the automated testing tool is insufficient? Is such a system in the scope of WCAG-ACT?

Thanks.

[WilcoFiers]

Hi Matt,

There are no open source tools that do what you ask. But Deque has a product called Assure which does what you describe.

Accessibility testing like that is in scope for ACT. The auto-WCAG group is wurging on rijles like that.

Hope that helps!

On 14 Dec 2016 17:03, "Matt Mikitka" notifications@github.com wrote:

@WilcoFiers https://github.com/WilcoFiers Thank you so much -- the JSON data will be very helpful.

Now, I can do the following outside of aXe:

1. Map aXe rules and checks to the Success Criteria and/or the Techniques in the W3C quickref JSON data.
2. Assign automated, semi-automated, manual methodology states to each sufficient criteria; if a methodology state is semi-automated or manual, then include steps for human intervention

@WilcoFiers https://github.com/WilcoFiers are you aware of existing test systems/services that provide the ability to distinguish automated checks from manual checks for a given automated accessibility testing tool, and prompt the user for human intervention wherever the automated testing tool is insufficient? Is such a system in the scope of WCAG-ACT?

Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dequelabs/axe-core/issues/261#issuecomment-267073783, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgY_z3Y4uuw4urANONeQGiJWT-wNCHaks5rIBNpgaJpZM4LC2WY .

[marcysutton]

I'm going to close this issue since it sounds outside of the scope of axe-core. Thanks everyone for the helpful discussion!

[yhtang68]

Hello, I have further questions on this. 1) How could we indicate WCAG 2.0 A/AA Rules and WCAG 2.1 A/AA Rules together? Could we document this usage in (https://www.deque.com/axe/axe-for-web/documentation/api-documentation/) e.g. I like to have

WCAG 2.0 Guidelines 1.1.1 Non-text Content - Level A 1.2.1 Audio-only and Video-only (Prerecorded) - Level A 1.2.2 Captions (Prerecorded) - Level A 3.1.2 Language of Parts - Level AA 3.2.1 On Focus - Level A 3.2.2 On Input - Level A 3.2.3 Consistent Navigation - Level AA 3.2.4 Consistent Identification - Level AA AND WCAG 2.1 Guidelines 1.3.4 Orientation - Level AA 1.3.5 Identify Input Purpose - Level AA 1.4.10 Reflow - Level AA 1.4.11 Non-Text Contrast - Level AA

2) What WCAG Rules are currently can NOT be tested via aXe? that required to be manually tested, e.g. Screen Reader?

3) How we may apply the rules defined in 1) when using aXe browser plugin? (https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)

Thank you Andy