AxeBuilder violation results are not consistent in @axe-core/playwright

[youvegotnigel]

Describe the bug I was experimenting with axe-core and playwright for accessibility testing in our organization and noticed this stage behavior and wanted to figure out what was going on here.

To Reproduce Steps to reproduce the behavior:

1. When I execute below code in playwright I get only one violation for {"description": "Ensures that lists are structured correctly"

test("CDC Dengue page should have only two WCAG A or AA violations", async ({page}, testInfo) => { await page.goto(https://www.cdc.gov/dengue/about/index.html`);

// wait for content to load
await page.waitForLoadState("domcontentloaded");

const accessibilityScanResults = await new AxeBuilder({ page })
  .withTags(["wcag2a", "wcag2aa"])
  .analyze();

await testInfo.attach("accessibility-scan-results", {
  body: JSON.stringify(accessibilityScanResults, null, 2),
  contentType: "application/json",
});

expect(accessibilityScanResults.violations).toHaveLength(2);

});`

2. But when I execute the same test but only added withRules(["skip-link"]), I still only get one violation and this time it is [{"description": "Ensure all skip links have a focusable target",

test("CDC Dengue page should have only two WCAG A or AA violations with skip link rule", async ({page}, testInfo) => { await page.goto(https://www.cdc.gov/dengue/about/index.html`);

// wait for content to load
await page.waitForLoadState("domcontentloaded");

const accessibilityScanResults = await new AxeBuilder({ page })
  .withTags(["wcag2a", "wcag2aa"])
  .withRules([
     "skip-link"
  ])
  .analyze();

await testInfo.attach("accessibility-scan-results", {
  body: JSON.stringify(accessibilityScanResults, null, 2),
  contentType: "application/json",
});

expect(accessibilityScanResults.violations).toHaveLength(2);

}); `

Expected behavior In the second test I should have got two violations.

1. Ensures that lists are structured correctly
2. Ensure all skip links have a focusable target

Environment :

• OS: Windows 10
• Playwright version: 1.45.0
• axe-core/playwright version: 4.9.1

Additional context

• Full error for test no 1:

  Error: expect(received).toHaveLength(expected)

  Expected length: 2 Received length: 1 Received array: [{"description": "Ensures that lists are structured correctly", "help": "

  and
  must only directly contain
  1. , Githubissues.
  2. Githubissues is a development platform for aggregating issues.