Violations count from axe-webdriverjs module and axe chrome extension is not same

[Hari2Fonteva]

Implemented this axe-webdriverjs node module in my automation framework (I attached the logic below, please take a look). We ran a scenario on one page and compared the violations count b/w the axe chrome extension and axe-webdriverjs node module but the count is not same. I am wondering why it's not giving the same results. Did anyone face this issue? Any suggestions, please?

With Axe chrome extension = 14 violations With axe-webdriverjs node module = 7 violations

axe chrome extension violations count report:

axe-webdriverjs node module violations count report:

Code Logic

import * as  AxeBuilder from 'axe-webdriverjs';

   public static async wcagCheck(): Promise<any> {
       return new Promise(async (resolve, reject) => {
           AxeBuilder(browser).options(
           {
           tags: { 'wcag2a': true, 'wcag2aa': true, 'section508': false }
            }).analyze(async (err, results) => {
              if (err) {
              reject(err);
              }
                 let currentTitle = await browser.getTitle(); // for report 
                 await fs.appendFile((path.join(__dirname, '/..') + 
                `/reports/json/violations/${currentTitle}_violations.json`), JSON.stringify(results));
           resolve({
          'url': results.url,
          'count': results.violations.length
        });
        // expect(results.violations.length, 'verify the violations length').to.equal(0);
      });
    });
  }

[straker]

Do you have a public page I could test this on? They should give the same results as both the extension and webdriver test the page with axe-core.

However, from an initial guess, I would say one of two things could have happened.

1) you are configuring the tags to run only wcag2a and wcag2aa, and not section508, while the chrome extension is set up to run all tags except experimental (so best-practice, section508, wcag21a, and wcag21aa). This is most likely the main cause of differences in violation numbers. 2) the page is slightly different in your images. The checkout sidebar has a yellow bar in one and a grey in the other. If there is text in that bar it could be color-contrast differences as well.

[Hari2Fonteva]

@straker: Thanks for the reply. I will check the above points and try to run it on a public page. I will share the status with you soon by adding/modifying the two points.

[Hari2Fonteva]

@straker: I tested this issue on the public page i.e., https://www.deque.com/axe/. Still, I am getting the same issue. If you get time tomorrow please let me know I will share more details. we will connect in zoom. Thanks.

axe-webdriverjs node module violations count report:

axe chrome extension violations count report:

Added 'section508': true , 'best-practice': true to function

public static async wcagCheck(): Promise<any> {
    return new Promise(async (resolve, reject) => {
      AxeBuilder(browser).options(
        {
          tags: { 'wcag2a': true, 'wcag2aa': true, 'section508': true , 'best-practice': true}
        }).analyze(async (err, results) => {
          if (err) {
            reject(err);
          }
          let currentTitle = await browser.getTitle();
          await fs.appendFile((path.join(__dirname, '/..') +
            `/reports/json/violations/${currentTitle}_violations.json`), JSON.stringify(results));
          resolve({
            'url': results.url,
            'count': results.violations.length
          });
          // expect(results.violations.length, 'verify the violations length').to.equal(0);
        });
    });
  }

[straker]

If you run the builder without passing any options, are the results still different?

[straker]

Ok, so I believe what's going on is just a difference in how the extension tallies violations. When a violation is returned, we list every node that failed that rule (each violation is a single rule). So the Unique ID rule is a single violation (out of 3), but has 12 related nodes. So to get the totals to match, you need to tally the nodes under each violation:

total = results.violations.reduce((acc, violation) => {
  return acc + violation.nodes.length;
}, 0);

console.log('total:', total);  // 15 (12 violations and 3 best practices)

The extra best practice violation is from the cookie consent popup which only appears when you haven't accepted it before.

[Hari2Fonteva]

@straker :

axe-webdriverjs: I validated your above logic on 'https://www.deque.com/axe/'. Without best-practices, we already have 15 nodes in the report. Looks like best-practices are separate tags in the violations Array. In your above logic, you are only checking the nodes count. So, it's showing the number 15. According to your logic if we count the violations nodes and best practices. The total count should be 15 violations nodes + 2 best-practice tags = 17. Check the below image.

axe chrome extension

Here we had total issues of 14 only. it's even not matching to your above logic of total 15 (12 violations and 3 best practices).

Click here to view JS file Can you share me more details please.

[Hari2Fonteva]

[Hari2Fonteva]

@straker : If you get time please check my above messages

[straker]

I'm not sure I understand, everything matches in your pictures. The plugin shows 14 total issues (All Issues) - 12 nodes that fail the unique-id rule and 2 nodes that fail the empty-heading rule. The output of the webdriver count shows 15 total, 12 nodes for unique-id and 2 nodes for empty-heading. The extra failure node is from the cookie consent modal as I said.

[Hari2Fonteva]

@straker : extra failure node is from the cookie consent modal - In my framework all features files are running in incognito window and everything time it will open new browser session and also we are clearing the cookies and caches before and after the sessions. I don't see the cookie modal to handle it. Can you please share me more info on how to handle this cookie consent modal. Thanks

[straker]

The only think I can think of is to click the consent button before you run axe (if it's present as it doesn't always seem to be). Other than that I'm not sure.

I'm going to close this issue as the original concern seems to have been addressed.