search

dequelabs / axe-core-gems

Ruby integration for axe-core, the accessibility testing engine
https://deque.com/axe
Mozilla Public License 2.0
88 stars 30 forks source link

Lots of test flake when upgrading from 4.7.0 to 4.8.0. #349

Closed mockdeep closed 9 months ago

mockdeep commented 12 months ago 

- `axe-core-rspec` version: 4.8.0
- Browser: Firefox

I'm having trouble figuring out why we're seeing this issue. Even when logging in our users, it will occasionally report accessibility violations like this:

Found 2 accessibility violations:

1) landmark-one-main: Document should have one main landmark (moderate)
    https://dequeuniversity.com/rules/axe/4.8/landmark-one-main?application=axeAPI
    The following 1 node violate this rule:

        Selector: html
        HTML: <html><head></head><body></body></html>
        Fix all of the following:
        - Document does not have a main landmark

2) page-has-heading-one: Page should contain a level-one heading (moderate)
    https://dequeuniversity.com/rules/axe/4.8/page-has-heading-one?application=axeAPI
    The following 1 node violate this rule:

        Selector: html
        HTML: <html><head></head><body></body></html>
        Fix all of the following:
        - Page must have a level-one heading

Invocation: axe.run({:exclude=>[]}, {:rules=>{"document-title"=>{:enabled=>false}, "html-has-lang"=>{:enabled=>false}}}, callback);

Prior to running a11y checks, we assert that we're fully on the following page. E.g.:

fill_in('session_email', with: user.email) 
fill_in('session_password', with: user.password)
click_button('Log in')
expect(page).to have_content('Manage')
expect(page).to be_axe_clean

However, it looks like axe-core is checking against a blank page, even though we've asserted that the page is loaded. These are server-rendered Rails pages without any client-rendering. There's a bunch of other JS loading on the page, but it doesn't do much on the home screen.

There was a little flake on 4.7.0, but we found that we could retry the a11y checks after a couple of seconds and they would work. Unfortunately, the retry logic doesn't resolve the issue on 4.8.0. It retries several times and still fails.

mockdeep commented 12 months ago

Extra baffling, we added an extra check to make sure content is rendered before checking a11y, and it still fails as if the page is blank:

within_window(current_window) do
  expect(page).to have_css('div')
  expect(page).to be_axe_clean
end
straker commented 11 months ago

First, some relevant comments from the axe-core issue

This seems to happen most often early in the test while logging in the user, and less so later in the spec. One thing I'm wondering is whether calling be_axe_clean in rapid succession might cause something like this. We call it on every page visit and button click. Is there any caching or asynchronous behavior in axe that might have a lingering affect? For more context, our login method looks like this:

visit(signin_path)
expect(page).to have_button('CONTINUE')
expect(page).to be_axe_clean
fill_in('session_email', with: user.email)

click_button('CONTINUE')
expect(page).to have_button('LOG IN')
expect(page).to be_axe_clean
fill_in('session_password', with: user.password)

click_button('LOG IN')
expect(page).to have_content('MANAGE')
expect(page).to be_axe_clean

What's odd is that we see the failure most frequently on the 3rd be_axe_clean call. Is it possible there is some lingering logic or caching from the previous calls?

straker commented 11 months ago

Continuing the discussion from the axe-core issue:

We did recently encounter a Java bug where how we were opening a blank page for the axe-core audit caused issues, so we'd like to run some experiments in Ruby to see if the same problem occurs outside of Java.

We're also wondering if this might be a browser driver issue. Did you per-chance upgrade your browser driver version at the same time you updated axe-core? If so could you try downgrading the browser driver package to what it was before to see if that makes the tests less flakey?

Also, when the log in button is clicked, what is the flow that happens? A lot of log in flows will perform redirects and it could be that with axe-core being faster as tests were turned off that it's trying to run axe-core during a redirect flow where the page it's currently on is currently blank. It would depend on when the "Manage" content is there and what happens after that.

Lastly, would it be possible to create a reproducible test case that we could test on our end? That would better help us debug the issue and make sure it's not an issue with our ruby library.

mockdeep commented 11 months ago

@straker thanks for the followup.

Did you per-chance upgrade your browser driver version at the same time you updated axe-core?

No, we've been on selenium-webdriver version 4.10.0 since June.

Also, when the log in button is clicked, what is the flow that happens?

Each step is a form post that redirects to the next page, but the assertions ensure the redirect is complete before running the accessibility checks. The "Manage" content shows up after the final redirect is completed.

Lastly, would it be possible to create a reproducible test case that we could test on our end?

Unfortunately, I don't know how to accomplish this. I'd be happy to do a screen share with you and walk you through things on our end. I did set it up on an open source project and put it on a repeat on CI, but so far haven't seen any flake. I'm a little surprised, actually, because on that project I haven't even added any assertions before running accessibility checks.

michael-siek commented 9 months ago

Hey @mockdeep just following up to see if this is still an issue.

mockdeep commented 9 months ago

Hey @michael-siek we've added a requirement to ensure that the next page is fully loaded after navigation and the amount of flake seems to have gone down significantly. In previous versions we were able to rescue from the various errors that popped up from axe-core and retry after a delay, but with the most recent version it almost seems like running the a11y checks causes navigation to halt, so retrying no longer works. Adding a required matcher makes sure the page is fully loaded, so I don't think there will be any more need for the retry.

I'll close this for now and I can report back in if anything else comes up.