Results consistency between webdriverjs CLI and browser extension?

[soykje]

Hello everyone! :

Product: webdriverjs

Expectation: We would expect to have same results between Axe browser extension and CLI (here webdriverjs).

Actual: I'm actually trying to achieve a little script to check accessibility on a Storybook project I'm working on (an UI components library). I already use the Axe browser extension (for Firefox). I wrote (based on documentation/examples) a simple first version of this script:

#!/usr/bin/env node

import AxeBuilder from '@axe-core/webdriverjs'
import { Builder } from 'selenium-webdriver'

import { Options as FirefoxOptions } from 'selenium-webdriver/firefox.js'

const driver = new Builder()
  .forBrowser('firefox')
  .setFirefoxOptions(new FirefoxOptions().headless())
  .build()

const url = 'https://sparkui.vercel.app/iframe.html?viewMode=docs&id=components-button--docs'

driver
  .get(url)
  .then(() => {
    new AxeBuilder(driver)
      .withTags(['best-practice', 'wcag21aa'])
      .options({ resultTypes: ['violations'], reporter: 'no-passes' })
      .analyze((_, results) => {
        const totalIssues = results.violations.reduce((acc, cur) =>
          acc += [...cur.nodes].length,
          0
        )

        console.log('Total Issues: ' + (totalIssues))

        results.violations.forEach(issue => {
          console.log('- ' + issue.description + ' (' + issue.nodes.length + ')')
        })
      })
  })

My problem is that with the browser extension I get 52 issues, but according to this script I end with 2... or 15!

vs

I already tried to finetune my configuration, but the biggest problem I have for now seems to be the inconsistency of those results (sometimes 2, sometimes 15, ...). Do you see any possible errors or mistakes within my script above, or could it be related to something else?

Thx in advance for your help! :pray:

@axe-core/webdriverjs: 4.7.2

- Node version: 18.16.0
- Platform: Linux

[Zidious]

Hey @soykje,

I am looking into this at the moment and will share results momentarily

[Zidious]

So, few things I've noticed; we analyse the page as soon as its ready. Launching in headed mode, you can see the page finishes loading then loads the component button docs, so it is only capturing the initial page load results, not the button documentation.

Additionally, you appear to only be running best-practice and WCAG21AA tags. The extension WCAG21AA encompasses the following tags: WCAG2A, WCAG2AA, WCAG21A and WCAG21AA.

To solve for all of the above, you can do something similar to the example below, I've modified your example slightly:

const AxeBuilder = require("@axe-core/webdriverjs");
const { Builder, By, until } = require("selenium-webdriver");
const { Options } = require("selenium-webdriver/firefox.js");

(async () => {
  const driver = await new Builder()
    .forBrowser("firefox")
    .setFirefoxOptions(new Options().headless())
    .build();

  const url =
    "https://sparkui.vercel.app/iframe.html?viewMode=docs&id=components-button--docs";

  await driver.get(url);

  // Wait for the Button H1 header to be visible
  await driver.wait(until.elementLocated(By.css("#button")), 10000);

  const results = await new AxeBuilder(driver)
    .withTags(["best-practice", "wcag2a", "wcag2aa", "wcag21a", "wcag21aa"])
    .options({ resultTypes: ["violations"], reporter: "no-passes" })
    .analyze();

  const totalIssues = results.violations.reduce(
    (acc, cur) => (acc += [...cur.nodes].length),
    0
  );

  console.log("Total Issues: " + totalIssues);

  results.violations.forEach((issue) => {
    console.log("- " + issue.description + " (" + issue.nodes.length + ")");
  });
})();

This will get you the same amount of results as the extension, you may need to tweak with what element you want to for to load as it appears they are loading after the the ready-state of the DOM.

[soykje]

@Zidious Thx a lot, you were right it works way better with this snippet! If I may, there is something I don't perfectly understand:

Additionally, you appear to only be running best-practice and WCAG21AA tags. The extension WCAG21AA encompasses the following tags: WCAG2A, WCAG2AA, WCAG21A and WCAG21AA.

If the WCAG21AA encompasses the other, why adding them to the withTags? Anyway, again thx a lot for you're help! :pray:

[Zidious]

By passing just WCAG21AA you are running a subset of rules that fall under WCAG2.1, in this case the rules that fall under WCAG2.1 AA of which axe-core has 1.

The extension works slightly differently, it runs all the rules up to WCAG21AA (excluding AAA rules). This means that will run the axe-core rules that fall under WCAG2A, WCAG2AA and WCAG21A, WCAG21AA criterions of which there are many.

[soykje]

Hmmmm... ok then! Thx for all these informations, you've been so helpful!!! Thx a lot :pray: