search

teamcapybara / capybara

Acceptance test framework for web applications
http://teamcapybara.github.io/capybara/
MIT License
10.01k stars 1.45k forks source link

Occasional `map!` errors when finding elements #2435

Closed mockdeep closed 3 years ago

mockdeep commented 3 years ago

Meta

Capybara Version:

3.34.0, though we were also seeing the issue on 3.29.0

Driver Information (and browser if relevant):

selenium-webdriver 3.142.7 with Firefox 84.0.1

Expected Behavior

Test passes consistently.

Actual Behavior

We occasionally see our test fail somewhere deep inside of Capybara. It's only a single test that is failing on a server rendered page without any JS interactions.

Stack Trace

     NoMethodError:
       undefined method `map!' for nil:NilClass
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/selenium/extensions/find.rb:31:in `gather_hints'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/selenium/extensions/find.rb:22:in `find_by'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/selenium/extensions/find.rb:11:in `find_css'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/node/base.rb:107:in `find_css'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/queries/selector_query.rb:243:in `find_nodes_by_selector_format'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/queries/selector_query.rb:158:in `block in resolve_for'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/node/base.rb:77:in `synchronize'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/queries/selector_query.rb:157:in `resolve_for'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/node/matchers.rb:844:in `block in _verify_selector_result'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/node/base.rb:83:in `synchronize'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/node/matchers.rb:843:in `_verify_selector_result'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/node/matchers.rb:110:in `assert_selector'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/session.rb:768:in `block (2 levels) in <class:Session>'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/rspec/matchers/have_selector.rb:18:in `element_matches?'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/capybara-3.34.0/lib/capybara/rspec/matchers/base.rb:51:in `matches?'
     # ./spec/features/admin/page_mover_spec.rb:50:in `block (2 levels) in <top (required)>'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:285:in `block in load'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:257:in `load_dependency'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:285:in `load'
     # /home/fletch/.rvm/gems/ruby-2.6.6/gems/spring-commands-rspec-1.0.4/lib/spring/commands/rspec.rb:18:in `call'

Steps to reproduce

Our test file:

RSpec.describe 'steps show page', :slow, js: true do

  let(:user) { create(:user, :admin) }
  let(:doc) { create(:doc) }

  before do
    create(:page, :with_image, doc: doc, seq: 1)
    page_2 = create(:page, :with_image, doc: doc, seq: 2)
    page_3 = create(:page, :with_image, doc: doc, seq: 3)

    create_list(:text_field, 5, page: page_2)
    create(:text_field, page: page_3)

    user_logs_in_as(user)
  end

  it 'allows IK12 admins to move pages [flaky]' do
    visit admin_doc_pages_path(doc), accessible: false

    pages = all('.page-in-list', count: 3)
    expect(pages[0]).to have_content('0 fields')
    expect(pages[1]).to have_content('5 fields')
    expect(pages[2]).to have_content('1 field')

    # Select the first page
    pages[0].click
    expect(find('.page-in-list.selected')).to have_content('0 fields')

    # Select the third page
    all('.page-in-list')[2].click
    expect(find('.page-in-list.selected')).to have_content('1 field')

    # Move the third page to an invalid seq
    fill_in('page[seq]', with: 8)
    click_button('Move', accessible: false)
    expect(page).to have_css('#page_seq:invalid')

    # Move the third page to seq 2
    fill_in('page[seq]', with: 2)
    click_button('Move', accessible: false)
    expect(page).to have_content('Page 3 moved to page 2')
    second_page = all('.page-in-list', count: 3)[1]
    expect(second_page).to have_content('1 field')
    expect(second_page[:class]).to include('selected')

    # Delete the (recently moved) second page
    accept_alert { click_button('Delete page and fields', accessible: false) }
    ######################################################
    expect(page).to have_selector('.page-in-list', count: 2) # line throwing the error
    ######################################################
    pages = all('.page-in-list')
    expect(pages[0]).to have_content('0 fields')
    expect(pages[1]).to have_content('5 fields')
  end

end
gurgelio commented 3 years ago

Same here, i've been shuffling versions all day but can't make it work. The first one or two examples get through, but then they all fail the same way

when running with selenium_headless:

Got 0 failures and 2 other errors:
  9.1) Failure/Error: click_on "Retificar Nome"
    NoMethodError:
      undefined method `map' for nil:NilClass
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/w3c/bridge.rb:561:in `find_elements_by'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/search_context.rb:80:in `find_elements'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/selenium/extensions/find.rb:17:in `find_by'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/selenium/extensions/find.rb:7:in `find_xpath'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/base.rb:116:in `find_xpath'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/queries/selector_query.rb:249:in `find_nodes_by_selector_format'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/queries/selector_query.rb:158:in `block in resolve_for'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/base.rb:77:in `synchronize'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/queries/selector_query.rb:157:in `resolve_for'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/finders.rb:294:in `block in synced_resolve'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/base.rb:83:in `synchronize'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/finders.rb:292:in `synced_resolve'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/finders.rb:53:in `find'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/actions.rb:26:in `click_link_or_button'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/session.rb:768:in `block (2 levels) in <class:Session>'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/dsl.rb:58:in `block (2 levels) in <module:DSL>'
    # ./spec/features/cursos/gerenciar_cursos_spec.rb:150:in `block (4 levels) in <top (required)>'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/webmock-3.11.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  9.2) Failure/Error: Unable to find WebDriverError@chrome://marionette/content/error.js to read failed line
    Selenium::WebDriver::Error::NoSuchAlertError:
    # WebDriverError@chrome://marionette/content/error.js:181:5
    # NoSuchAlertError@chrome://marionette/content/error.js:382:5
    # GeckoDriver.prototype._checkIfAlertIsPresent@chrome://marionette/content/driver.js:3476:11
    # GeckoDriver.prototype.dismissDialog@chrome://marionette/content/driver.js:3379:8
    # despatch@chrome://marionette/content/server.js:297:40
    # execute@chrome://marionette/content/server.js:267:16
    # onPacket/<@chrome://marionette/content/server.js:240:20
    # onPacket@chrome://marionette/content/server.js:241:9
    # _onJSONObjectReady/<@chrome://marionette/content/transport.js:504:20
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:72:in `assert_ok'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:34:in `initialize'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `new'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `create_response'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/default.rb:114:in `request'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:64:in `call'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:167:in `execute'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/w3c/bridge.rb:567:in `execute'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/w3c/bridge.rb:173:in `window_handles'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:187:in `window_handles'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/selenium/driver.rb:233:in `window_handles'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/selenium/driver_specializations/firefox_driver.rb:54:in `reset!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/session.rb:130:in `reset!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara.rb:325:in `block in reset_sessions!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara.rb:325:in `reverse_each'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara.rb:325:in `reset_sessions!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/rspec.rb:20:in `block (2 levels) in <top (required)>'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/webmock-3.11.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

when running on selenium_headless_chrome:

Got 0 failures and 2 other errors:
  8.1) Failure/Error: click_on "Descredenciar Curso"
    Capybara::ElementNotFound:
      Unable to find link or button "Descredenciar Curso"
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/finders.rb:303:in `block in synced_resolve'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/base.rb:83:in `synchronize'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/finders.rb:292:in `synced_resolve'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/finders.rb:53:in `find'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/node/actions.rb:26:in `click_link_or_button'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/session.rb:768:in `block (2 levels) in <class:Session>'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/dsl.rb:58:in `block (2 levels) in <module:DSL>'
    # ./spec/features/cursos/gerenciar_cursos_spec.rb:55:in `block (4 levels) in <top (required)>'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/webmock-3.11.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  8.2) Failure/Error: e = error
    Selenium::WebDriver::Error::InvalidArgumentError:
      invalid argument: 'name' must be a string
        (Session info: headless chrome=87.0.4280.88)
        (Driver info: chromedriver=87.0.4280.88 (89e2380a3e36c3464b5dd1302349b1382549290d-refs/branch-heads/4280@{#1761}),platform=Linux 5.8.0-7630-generic x86_64)
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:72:in `assert_ok'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:34:in `initialize'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `new'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `create_response'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/default.rb:114:in `request'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:64:in `call'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:167:in `execute'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/oss/bridge.rb:587:in `execute'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/oss/bridge.rb:154:in `window_handles'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:187:in `window_handles'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/selenium/driver.rb:233:in `window_handles'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/selenium/driver_specializations/chrome_driver.rb:41:in `reset!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/session.rb:130:in `reset!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara.rb:325:in `block in reset_sessions!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara.rb:325:in `reverse_each'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara.rb:325:in `reset_sessions!'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/capybara-3.34.0/lib/capybara/rspec.rb:20:in `block (2 levels) in <top (required)>'
    # /home/gurgel/.rvm/gems/ruby-2.5.3/gems/webmock-3.11.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'
twalpole commented 3 years ago

A quick fix for this is to set the DISABLE_CAPYBARA_SELENIUM_OPTIMIZATIONS environment variable which will disable the speed optimizations being done for collecting attributes for multiple elements during find. Beyond that fixing the error is simple although I would like to know how this error is ever happening in the first place in order to add tests for it. Are these very dynamic pages where matching elements would be disappearing while gather_hints is running? You're looking for a total of 2 matching elements, but the optimizations are only ever used if there are more than 2 elements returned by the initial find which may imply a page where potentially matching elements are being removed? Also accessible: false isn't a valid Capybara option, so what are you using that is potentially patching Capybara?

twalpole commented 3 years ago

@LeoGurja Your error(s) are not the same as you can see by looking at the stack traces, and appear to be that expected things aren't actually appearing on the page. If you believe those elements are actually there please file a new issue with a way to replicate the problem you're seeing.

twalpole commented 3 years ago

@mockdeep Please try the issue_2435 branch and see if it fixes the issue you're having

gurgelio commented 3 years ago

@twalpole Oh sorry, that's what I get when running capybara 3.34, when running 3.15 I get the same undefined 'map'. Different version combinations yield different error messages, I can't find any combination that works. I think it's a dependency hell problem because it started after an apt upgrade. All examples that used to succeed are now failing

capybara (3.34.0) selenium-webdriver (3.142.7) chromedriver (87.0.4280.88) - downloaded with the webdrivers gem (4.4.2)

gurgelio commented 3 years ago

I managed to find my problem. Yesterday I was so tired I couldn't even think straight, not every test is failing, some tests in my suite are causing Selenium to timeout. After that, it crashes and everything gets weird with undefined names and maps everywhere. I don't my errors and @mockdeep 's are the same I am trying to fix it now:

Failure/Error: active += page.evaluate_script("$.active")
  Timeout::Error:
    execution expired

I won't go into details about it because it's not important for this opened issue

As for @mockdeep, I would recommend trying the default driver (RackTest) since it doesn't involve any JS

twalpole commented 3 years ago

@LeoGurja Yeah - unfortunately when you go against suggested best practices (test visual page changes, not JS level properties) - things can go very wrong. Timeout::timeout is one of the most dangerous Ruby methods since when it timeouts it cancels threads without notice and therefore can leave things like network stacks in completely unknown states. As to why the JS "$.active" could ever timeout that possibly sounds like an issue with the latest chromedriver.

gurgelio commented 3 years ago

I removed my wait_for_ajax helper. It is no longer timing out. I had no idea Capybara had built-in retries, making this helper actually useless 🤡 There was a problem with it, it was not incrementing and decrementing request count correctly

twalpole commented 3 years ago

I've merged the "fix" into master, and am going to close this. Would still appreciate info of what exactly the page is doing at that moment in time.

mockdeep commented 3 years ago

@twalpole sorry for the delay. Thanks for getting a fix out so quickly. We'll test it out in the next couple of days and let you know if it does the trick.

The page is server rendered and as far as I've seen there's no JS running on it, aside from Rails UJS. Clicking the delete button redirects back to the index page, and goes from showing 3 elements to 2. The failure screen grab shows 2 elements on the page. The one thing that stands out as potentially interesting is that there are images in each of the elements, so maybe those aren't fully loaded at the time Capybara is trying to do it's thing?

mockdeep commented 3 years ago

Okay, @twalpole I ran the tests continuously for a couple of days based on your fix and didn't see that one fail again. As another alternative, I tried adding an assertion on the flash message after the accept_alert and the spec also stopped being flaky:

  accept_alert { click_button('Delete page and fields', accessible: false) }
+ expect(page).to have_content('Page deleted!')
  expect(page).to have_selector('.page-in-list', count: 2)

Maybe there's some combination of accept_alert and asserting against something that is on both pages that triggers the issue? In case it helps, the surrounding html structure looks like this:

<div class="row page-list">
  <div class="page-in-list wb-u-margin-v-1 wb-u-margin-h-1 ">
    <a href="/admin/docs/466243787/pages/199">
      <img id="img-199" src="/uploads/doc/paper/199/thumb_img_permission_1.png">
    </a>
    <div class="wb-l-center page-info">1</div>
    <div class="wb-l-center field-info">
      0 fields
    </div>
  </div>
  <div class="page-in-list wb-u-margin-v-1 wb-u-margin-h-1 ">
    <a href="/admin/docs/466243787/pages/200">
      <img id="img-200" src="/uploads/doc/paper/200/thumb_img_permission_2.png">
    </a>
    <div class="wb-l-center page-info">2</div>
    <div class="wb-l-center field-info">
      5 fields
    </div>
  </div>
</div>