Reporting AXE scores

[nniiicc]

A few resources to get started - in particular see

• https://www.deque.com/axe-con/wp-content/uploads/2021/12/axe-con_Accessibility_Scoring_v1.0_20220303.pdf

• https://www.matuzo.at/blog/building-the-most-inaccessible-site-possible-with-a-perfect-lighthouse-score/

• Fleisch Kincaid - scored by reading grade level

• https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests

[peiwenf]

Some other readings:

• axe-core Github repo
  • https://github.com/dequelabs/axe-core
• description of the axe.run()
  • https://www.deque.com/axe/core-documentation/api-documentation/#api-name-axerun
• how axe mobile calculates the accessibility score
  • https://docs.deque.com/devtools-mobile/2022.11.17/en/score
    
• Other ways to test accessibility
  • https://www.smashingmagazine.com/2021/06/complete-guide-accessibility-tooling/

[nniiicc]

Some things to look at for how Axe Scores are reported:

• https://www.sciencedirect.com/science/article/pii/S1319157818312394
• https://link.springer.com/article/10.1007/s10209-022-00866-y
• This chapter might be good for how the tools are described and how their data might be reported https://link.springer.com/chapter/10.1007/978-1-4471-7440-0_26

[nniiicc]

Axe rule mapped to WCAG

• https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md
• One thing it would be good to know is how A vs AA vs AAA are calculated / related to Axe scores

[nniiicc]

Placed to look for other papers:

• arXiv - https://arxiv.org/ (pre-print for CS papers)
• Google Scholar

Some things to search for:

• Axe scores / Axe reporting
• Accessibility monitoring / reporting

[peiwenf]

@nniiicc An overview for all possible metrics:

• https://dl.acm.org/doi/epdf/10.1145/1456536.1456551

some related but more in-depth articles: WAQM metrics(the metrics that considers warning):

• https://www.researchgate.net/publication/220933611_Quantitative_metrics_for_measuring_web_accessibility

tool comparison between WAVE and SiteImprove:

• https://www.researchgate.net/publication/338550460_Comparing_Web_Accessibility_Evaluation_Tools_and_Evaluating_the_Accessibility_of_Webpages_Proposed_Frameworks
  • mentions a metrics called RA-WAEM, where the score is calculated by aggregating the rate of the webpage passes the test over the website that don't pass the test here is a more in-depth talk about that https://dl.acm.org/doi/epdf/10.1145/3192714.3192836

Some real world apply: AudioEye

• https://audioeye.medium.com/accessibility-score-methodology-deep-dive-4e405e0f923c
  • page-level score and site-level score
  • page level score is calculated by aggregating the results of our tests into failure rates, then multiplying the failure rates by weight.
  • uses equal weighting and partial pass
  • 80 might be the threshold

Lighthouse

• https://developer.chrome.com/docs/lighthouse/accessibility/scoring/
• If the violation>0 for a test, then it's failed. Weight each test differently based on its impact level
• use priority weight and doesn't allow partial pass
  • The weight that light house uses is not depend on the axe report

Suggestions: With the current data, I think the easiest way to get a score is to follow a similar path as AudioEye. If we are trying to get around the parsing we can probably not allow the partial pass and replace possible tests per website with the total number of possible tests and use this calculation to get the fail rate: # of test which has a violation/ the total number of possible tests

Notes: AudioEye API is only open for partners: https://help.audioeye.com/hc/en-us/articles/4409625271700-API-Introduction

[peiwenf]

Metrics used(Using AudioEye's page score method to get a site score) Failed = the number of nodes under the violations category Tested = the sum of the nodes under violations, passes, and incomplete category Total = Total number of applicable tests Sum of all tests using this formula: (Failed/Tested)*(1/Total)