8.8.x issues with Behat tests and selenium/chromedriver

[dpagini]

My system information:

• BLT version: 8.8

I tried upgrading my project to the latest BLT 8.8.x branch (8.8.2) and was having major problems with my behat tests passing Travis CI. I know this new minor version starts moving away from Phing to Robo, particularly for tests, but that's the main thing that I can see that's different.

I am having a very similar issue to geerlingguy/drupal-vm#1152. The test suite stuff is a little over my head here, but I'm generally following that issue to say there's not a good solution right now. What I'm confused about is how/why this only happening in the 8.7 -> 8.8 upgrades, but my behat tests are running correctly now.

Anyways, I'm willing to help look into this issue more, and was just wondering if anyone else was having similar issues if they tried to upgrade...? If I can help provide any info, or try anything, please LMK.

Is there a reason BLT uses se/selenium-server-standalone 2.53 when there are newer versions available?

[grasmash]

@dpagini There isn't enough information here for me to really guess at the issue. What errors did you encounter? What's the log output? Does it only occur on Travis? Are you using Selenium or Phantom?

In general, Selenium and Chrome are not playing well together right now. This is an issue outside of BLT. The interim solution is to fallback to using PhantomJS, which isn't ideal because it tends to behave inconsistently on Travis.

We're planning to move to native headless Chrome once Behat is interoperable with it.

[dpagini]

There isn't enough information here for me to really guess at the issue. What errors did you encounter? What's the log output? Does it only occur on Travis? Are you using Selenium or Phantom?

Sorry, I should have figured as much. I didn't think this would be a simple fix so I just wanted to post about the problem and offer to help, but this is probably a better first step.

I'm using Selenium. The issue was plaguing me both on Travis and local on my Drupal VM. I sort of knew that the issue is bigger than BLT based on reading the ticket I linked to in my original post, BUT... my project on BLT 8.7.2 runs these same behat tests fine, so something about the upgrade to 8.8.2 is causing these errors for me. I was trying to figure out what else changed, and the piece I was left with was maybe something to do with Phing vs. Robo calls to behat? It seems like the relevant behat composer projects are the same versions, and my travis configuration didn't get upgraded at all...

Here is the behat errors from Travis:

tests:behat >
[info] Killing all processing containing string 'runserver'
[info] Killing all processes on port 8888
➜  Launching PHP's internal web server via drush.
[info] Running server at http://127.0.0.1:8888
[info] Waiting for response from http://127.0.0.1:8888...
[info] Waiting for response from http://127.0.0.1:8888...
[info] Waiting for response from http://127.0.0.1:8888...
[info] Waiting for response from http://127.0.0.1:8888...
[Filesystem\FilesystemStack] mkdir ["/home/travis/build/blt-project/reports"]
 1/1 [============================] 100% < 1 sec
[info] Creating Selenium2 log file at /home/travis/build/blt-project/reports/selenium2.log
[Filesystem\FilesystemStack] touch ["/home/travis/build/blt-project/reports/selenium2.log"]
[info] Killing any running Selenium processes
[info] Killing all processes on port 4444
[info] Killing all processing containing string 'selenium-server-standalone'
[info] Launching Selenium standalone server.
[info] Waiting for response from http://127.0.0.1:4444/wd/hub...
[info] Waiting for response from http://127.0.0.1:4444/wd/hub...
[Testing\Behat] Running behat  --format pretty /home/travis/build/blt-project/tests/behat/features/web --no-interaction --stop-on-failure --strict --config /home/travis/build/blt-project/tests/behat/local.yml --profile local --tags '~ajax&&~experimental&&~lightningextension'

[... truncated by @dpagini ...]

 Scenario: Create Article Content                                                          # features/web/content/article.feature:21
    Given I am logged in as a user with the "create article content" permission             # Drupal\DrupalExtension\Context\DrupalContext::assertLoggedInWithPermissions()
    And I have a file named "TestBehatImage.png"                                            # Drupal\FeatureContext::iHaveaFileNamed()
      │ 
      │  File found: /home/travis/build/Voya-IM/drupal-platform/tests/behat/media/TestBehatImage.png
      │ 
    Then I go to "node/add/article"                                                         # Drupal\DrupalExtension\Context\MinkContext::visit()
    And I select "Blog" from "Article type"                                                 # Drupal\DrupalExtension\Context\MinkContext::selectOption()
    And I fill in the following:                                                            # Drupal\DrupalExtension\Context\MinkContext::fillFields()
      | Title                                  | Article Behat Test |
      | Article tags                           | TestTag            |
      | field_publication_date[0][value][date] | 12/01/2016         |
      | Article author                         | Author Name        |
    And I fill in wysiwyg field "Article description" with "This is my article text"        # Drupal\FeatureContext::iFillInWysiwygFieldWith()
    When I click "Promo"                                                                    # Drupal\DrupalExtension\Context\MinkContext::assertClick()
    And I wait for AJAX to finish                                                           # Drupal\DrupalExtension\Context\MinkContext::iWaitForAjaxToFinish()
    And I check the box "Featured Content"                                                  # Drupal\DrupalExtension\Context\MinkContext::assertCheckBox()
    And I fill in "Short Title" with "Short title"                                          # Drupal\DrupalExtension\Context\MinkContext::fillField()
    And I attach the file "TestBehatImage.png" to "Hero Image"                              # Drupal\DrupalExtension\Context\MinkContext::attachFileToField()
    And I wait for AJAX to finish                                                           # Drupal\DrupalExtension\Context\MinkContext::iWaitForAjaxToFinish()
    And I fill in "field_hero_image[0][alt]" with "Alternative text for article hero image" # Drupal\DrupalExtension\Context\MinkContext::fillField()
    And I fill in wysiwyg field "Body" with "Lorem ipsum."                                  # Drupal\FeatureContext::iFillInWysiwygFieldWith()
    And I press "Save"                                                                      # Drupal\DrupalExtension\Context\MinkContext::pressButton()
    Then I should see "Article Article Behat Test has been created."                        # Drupal\DrupalExtension\Context\MinkContext::assertPageContainsText()
  │
  ╳  timeout: cannot determine loading status
  ╳  from timeout: Timed out receiving message from renderer: -0.006
  ╳    (Session info: chrome=55.0.2883.75)
  ╳    (Driver info: chromedriver=2.24.417424 (c5c5ea873213ee72e3d0929b47482681555340c3),platform=Linux 4.4.0-51-generic x86_64) (WARNING: The server did not provide any stacktrace information)
  ╳  Command duration or timeout: 300.01 seconds
  ╳  Build info: version: '2.53.1', revision: 'a36b8b1', time: '2016-06-30 17:37:03'
  ╳  System info: host: 'testing-gce-db2ac5a0-b187-4d63-9bae-94e552f23ee7', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-51-generic', java.version: '1.8.0_111'
  ╳  Driver info: org.openqa.selenium.chrome.ChromeDriver
  ╳  Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.24.417424 (c5c5ea873213ee72e3d0929b47482681555340c3), userDataDir=/tmp/.org.chromium.Chromium.1qJsyS}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=55.0.2883.75, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true}]
  ╳  Session ID: 09929302cd7b1509444b864fefcd6b42 (WebDriver\Exception\ScriptTimeout)

[dpagini]

...and on Drupal-VM, similar errors. These are the same tests as above, I am just cutting out some of the behat commands. Note that the error occurs earlier in the behat test on DrupalVM vs Travis, but it seems like it's a similar error.

$ blt tests:behat
[Testing\Behat] Running behat  --format pretty /var/www/drupal/tests/behat/features/web --no-interaction --stop-on-failure --strict --config /var/www/drupal/tests/behat/local.yml --profile local --tags '~ajax&&~experimental&&~lightningextension'

[... truncated by @dpagini ...]

  Scenario: Create Article Content                                                          # features/web/content/article.feature:21
    Given I am logged in as a user with the "create article content" permission             # Drupal\DrupalExtension\Context\DrupalContext::assertLoggedInWithPermissions()
      Could not open connection: unknown error: Chrome failed to start: exited abnormally
        (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 3.13.0-116-generic x86_64) (WARNING: The server did not provide any stacktrace information)
      Command duration or timeout: 60.23 seconds
      Build info: version: '2.53.1', revision: 'a36b8b1', time: '2016-06-30 17:37:03'
      System info: host: 'local', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-116-generic', java.version: '1.7.0_121'
      Driver info: org.openqa.selenium.chrome.ChromeDriver (Behat\Mink\Exception\DriverException)
      Could not open connection: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 3.13.0-116-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.04 seconds
Build info: version: '2.53.1', revision: 'a36b8b1', time: '2016-06-30 17:37:03'
System info: host: 'local', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-116-generic', java.version: '1.7.0_121'
Driver info: org.openqa.selenium.chrome.ChromeDriver

[...]

  │
  ╳  Could not open connection: unknown error: Chrome failed to start: exited abnormally
  ╳    (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 3.13.0-116-generic x86_64) (WARNING: The server did not provide any stacktrace information)
  ╳  Command duration or timeout: 60.05 seconds
  ╳  Build info: version: '2.53.1', revision: 'a36b8b1', time: '2016-06-30 17:37:03'
  ╳  System info: host: 'local', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-116-generic', java.version: '1.7.0_121'
  ╳  Driver info: org.openqa.selenium.chrome.ChromeDriver (Behat\Mink\Exception\DriverException)
  │
  └─ @AfterScenario # Drupal\DrupalExtension\Context\DrupalContext::cleanUsers()

--- Failed scenarios:

    features/web/content/article.feature:21

2 scenarios (1 passed, 1 failed)
35 steps (8 passed, 1 failed, 26 skipped)
3m12.70s (87.02Mb)
[Testing\Behat]  Exit code 1 
[error]  Behat tests failed 

[dpagini]

Also, my $ blt doctor output is giving me trouble with my checkUri parameter. I am currently trying to research that to see if it's a problem with my project or BLT. I dont know if it's related to this issue, but it was the one thing that BLT was reporting as a problem, so I was looking into fixing it. I do have my local site URL set in docroot/sites/default/local.drushrc.php in the $options['uri'] setting, but BLT thinks my site URI is set to "default".

[grasmash]

I think that the blt doctor issue is separate.

I would suggest debugging this by running the commands directly, without BLT. You'll need two tabs: one to run selenium server and one to run behat.

./vendor/bin/behat  --format pretty /var/www/drupal/tests/behat/features/web --no-interaction --stop-on-failure --strict --config /var/www/drupal/tests/behat/local.yml --profile local --tags '~ajax&&~experimental&&~lightningextension'

That should let you cut to the heart of it.

[dpagini]

Yep, sorry, I opened #1408 for the checkUri (hope it's valid). Thanks for the suggestion. I'll give that a try and report back as soon as I can.

[dpagini]

You'll need two tabs: one to run selenium server and one to run behat.

Can you explain this more @grasmash? Sorry, I guess I'm just a little more unfamiliar with selenium than I should be. Are you referring to command line "tabs"? How do I run the selenium server, and what does it do to run that separate? With DrupalVM, I'm running all these commands while I'm SSH'd into my VM, right?

Hmm... ok, anyways, I learned that my behat tests are failing on my VM with any version of BLT. I guess that's to your point earlier about selenium/chrome. I still know for a fact though that behat is passing in Travis with 8.7.2 but failing with 8.8.2, which is still leaving me scratching my head.

[grasmash]

Basically, BLT does two things when it executes a Behat test.

1. It launches a web driver as a background process. To do this manually, just open a CLI tab and run ./vendor/bin/selenium-server-standalone -port 4444 if you're using Selenium, or ./vendor/bin/phantomjs --webdriver 4444 if you're using PhantomJS.
2. It runs behat. You can do this manually by opening a second CLI tab (while Selenium/PhantomJS is running) and executing: ./vendor/bin/behat --format pretty ./tests/behat --no-interaction --stop-on-failure --strict --config ./tests/behat/local.yml --profile local --tags '~ajax&&~experimental&&~lightningextension'

If that doesn't work, then there's an issue outside of BLT. It could be:

1. Your Behat config. tests/behat/behat.yml and tests/behat/local.yml
2. Your Behat driver (behat/mink-selenium2-driver)
3. Your browser (chrome) and it's headless driver (chromedriver)
4. Selenium

At present, the latest version of Chrome and Chromedriver do not appear to be working well with Behat. See #1134 and #1126. Unfortunately this is outside of the scope of BLT and there is no known solution. The available workaround is to use PhantomJS rather than Selenium.

Behat is a bit difficult because it requires so many things working in concert. This should be much simpler once we have a headless chrome browser and we can remove PhantomJS/Selenium from the mix.

[geerlingguy]

Should this be marked a duplicate of #1126?

[dpagini]

I'm OK with that. I don't want to spend too much time figuring this out if we know there are selenium/chromedriver/etc issues. I'm mostly befuddled by the switch from Phing to Robo seemingly causing my selenium tests to start failing. I took your and @grasmash's advice though and moved over to PhantomJS. I actually had to re-write a few tests though to do that, oddly enough. I think I'm in an OK enough place for now, and hopefully one day I can switch back to selenium, but I would rather stay up to date in BLT.