window.axe is undefined

[mockdeep]

We see occasional flake in our builds where axe isn't loaded into the page for some reason. Maybe it isn't ready, yet?

Stack Trace

``` Minitest::UnexpectedError: UncaughtThrowError: uncaught throw {"errorMessage"=>"window.axe is undefined", "passes"=>[], "timestamp"=>"Wed May 04 2022 15:41:38 GMT-0700 (Pacific Daylight Time)", "url"=>"", "violations"=>[]} /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/axe-core-api-4.4.0/lib/axe/api/run.rb:120:in `throw' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/axe-core-api-4.4.0/lib/axe/api/run.rb:120:in `run_partial_recursive' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/axe-core-api-4.4.0/lib/axe/api/run.rb:36:in `analyze_post_43x' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/axe-core-api-4.4.0/lib/axe/core.rb:18:in `call' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/axe-core-api-4.4.0/lib/axe/matchers/be_axe_clean.rb:22:in `audit' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/axe-core-api-4.4.0/lib/axe/matchers/be_axe_clean.rb:26:in `matches?' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/handler.rb:51:in `block in handle_matcher' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/handler.rb:27:in `with_matcher' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/handler.rb:48:in `handle_matcher' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/expectation_target.rb:65:in `to' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/expectation_target.rb:101:in `to' /home/circleci/app/spec/support/accessibility_overrides.rb:19:in `click_button' /home/circleci/app/spec/features/doc_setup/auto_detect_fields_spec.rb:72:in `block (3 levels) in ' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.5.1/lib/active_support/testing/assertions.rb:34:in `assert_nothing_raised' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/activejob-6.1.5.1/lib/active_job/test_helper.rb:594:in `perform_enqueued_jobs' /home/circleci/app/spec/features/doc_setup/auto_detect_fields_spec.rb:71:in `block (2 levels) in ' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:263:in `instance_exec' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:263:in `block in run' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:486:in `block in run' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:626:in `block in run_around_example_hooks_for' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:352:in `call' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-rails-5.1.2/lib/rspec/rails/adapters.rb:75:in `block (2 levels) in ' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:457:in `instance_exec' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:457:in `instance_exec' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:390:in `execute_with' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:352:in `call' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:629:in `run_around_example_hooks_for' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:486:in `run' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:468:in `with_around_example_hooks' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:259:in `run' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:646:in `block in run_examples' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:642:in `map' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:642:in `run_examples' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:607:in `run' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:121:in `map' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/configuration.rb:2068:in `with_suite_hooks' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:116:in `block in run_specs' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/reporter.rb:74:in `report' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:115:in `run_specs' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:89:in `run' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:71:in `run' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:45:in `invoke' /home/circleci/app/vendor/bundle/ruby/3.0.0/gems/rspec-core-3.11.0/exe/rspec:4:in `' /home/circleci/app/vendor/bundle/ruby/3.0.0/bin/rspec:25:in `load' /home/circleci/app/vendor/bundle/ruby/3.0.0/bin/rspec:25:in `' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `load' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `kernel_load' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:23:in `run' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/cli.rb:479:in `exec' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/cli.rb:31:in `dispatch' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/cli.rb:25:in `start' /home/circleci/.rubygems/gems/bundler-2.2.33/exe/bundle:49:in `block in ' /home/circleci/.rubygems/gems/bundler-2.2.33/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors' /home/circleci/.rubygems/gems/bundler-2.2.33/exe/bundle:37:in `' /home/circleci/.rubygems/bin/bundle:25:in `load' /home/circleci/.rubygems/bin/bundle:25:in `

' ./spec/support/accessibility_overrides.rb:19:in `click_button' ./spec/features/doc_setup/auto_detect_fields_spec.rb:72:in `block (3 levels) in ' ./spec/features/doc_setup/auto_detect_fields_spec.rb:71:in `block (2 levels) in ' ------------------ --- Caused by: --- UncaughtThrowError: uncaught throw {"errorMessage"=>"window.axe is undefined", "passes"=>[], "timestamp"=>"Wed May 04 2022 15:41:38 GMT-0700 (Pacific Daylight Time)", "url"=>"", "violations"=>[]} ./spec/support/accessibility_overrides.rb:19:in `click_button' ```

- `axe-core-rspec` version: 4.4.0
- Firefox 97.0.2
- Capybara w/ RSpec

[michael-siek]

Hey @mockdeep just following up to see if this is still an issue.

[mockdeep]

@michael-siek I think this was probably due to the page not being fully loaded. We ended up implementing some retry logic that caught the error and retried with a backoff. This worked pretty well for a while, but we had this other issue with flakiness in version 4.8.0.

The workaround in our code was to ensure that the page was fully loaded with an extra assertion before doing a11y checks. For what it's worth, it seems like ideally axe-core-rspec would handle the possibility of flake related to page loading, similar to the way that Capybara does. Capybara provides a synchronize method where you can catch certain errors and have it retry until the timeout.

I'm fine closing this out, though, if you don't think it's appropriate.

[michael-siek]

@mockdeep thank you for the response! will close and can reopen if needed.

[padmavemulapati]

As gems being able to scan the page successfully and nothing is reproducing .

https://github.com/dequelabs/axe-core-gems/assets/47409579/37aac8d8-8556-408a-bc6b-03183b65b881

Environment:

Label	Value
Product	axe-core-gems/rspec-cucumber
Version	4.8.2
OS-Details	MAC- Intel Core i7 - 11.6.8 && Windows 11
BrowserDetails	Chrome Version 121.0.6167.139 && Firefox 122.0.1