Inconsistency when checking multiple URLs vs single

fredboyle commented 5 years ago

We work on a design system and check each component example individually for accessibility. However we're receiving false failures when running the axe check by passing an array of URLs vs checking the same failing URL as a single entry.

So when a bulk list of URLs is passed in certain ones report failures, however if you take one of those same URLs and check it on it's own or as the first one in a list then no failure is reported for that URL.

Any idea what might be causing this?

We are using the latest v3.1.3 of gulp-axe-webdriver and have axe-core v3.2.2 installed.

felixzapata commented 5 years ago

hi, those urls are from local dev servers or public? Does it happen always? As far as I know I didn't detect any issues related with that on my current tests. With public urls sometimes I discovered problems related with the load time due to ads and external resources.

fredboyle commented 5 years ago

@felixzapata Local dev server started by the automated test script. The issue occurs every time as described in my previous comment.

No ads or anything external, all resources are from the local dev server directly.

Any thoughts on why this might be occurring?

felixzapata commented 5 years ago

I would need to make more tests about that. Does it happen with some particular numbers of urls to check?

fredboyle commented 5 years ago

It occurs with as few as two URLs. If the URL is the first one it's fine but if it's the second or later it will show failures that shouldn't occur.

felixzapata commented 5 years ago

ok, I will try to reproduce it.

felixzapata commented 5 years ago

@fredboyle I can't reproduce the issue. I'm using this options:

gulp.task('axe14', function() {
  var options = {
            saveOutputIn: 'allHtml.json',
            urls: ['http://127.0.0.1:8887/working.html', 'test/fixtures/broken.html', 'http://127.0.0.1:8887/a.html', 'test/fixtures/working2.html'],
            headless: true
    };
    return axeWebDriver(options);
});

Captura de pantalla 2019-05-11 a las 16 05 20

Captura de pantalla 2019-05-11 a las 16 05 22

May I see your task options?

fredboyle commented 5 years ago

export default paths =>
  gulpAxe({
    folderOutputReport: '.reports',
    saveOutputIn: 'a11y.json',
    headless: true,
    a11yCheckOptions: {
      rules: {
        bypass: { enabled: false },
        'landmark-one-main': { enabled: false },
        region: { enabled: false },
        'page-has-heading-one': { enabled: false }
      }
    },
    exclude:
      // ...list of exclusions here,
    urls: paths, // dynamically generated list of http://localhost/ URLs
    showOnlyViolations: true,
    verbose: true,
    errorOnViolation: true
  });

felixzapata commented 5 years ago

can you test only with these options?: saveOutputIn, urls and headless.

fredboyle commented 5 years ago

No change with only those options. Order still matters and false failures still occur beyond the first URL.

felixzapata commented 5 years ago

can you try using only these two html files inside your server?

<!doctype html>
<html lang="es">
<head>
    <title>Test</title>
</head>
<body>
  <div id="working" role="main">
    <h1>Foobar tittle</h1>
    <label for="has-label">Label for this text field.</label>
    <input type="text" id="has-label">
  </div>
</body>
</html>

<!doctype html>
<html>
<head>
    <title>Test</title>
</head>
<body>
  <div id="broken" role="main">
    <h1>Foobar tittle</h1>
    <p>Not a label</p>
    <input type="text" id="no-label">
    <p>Not an alt</p>
    <img src="foobar.gif">
  </div>
</body>
</html>

fredboyle commented 5 years ago

Those two files generate the expected failures regardless of order.

Our issues appear to revolve primarily around contrast color checking. Values make no sense and when a false failure URL is first in line it properly passes as expected.

fredboyle commented 5 years ago

After much digging it appears the way our code was being generated and tested was part of the issue. Using a different approach without issues at this point. Thanks for the help.

felixzapata / gulp-axe-webdriver

Inconsistency when checking multiple URLs vs single #24