gulp e2e failures

[gghai]

C:\Users\builder\marklogic-samplestack>gulp e2e [18:16:47] Starting 'clean'... [18:16:47] Starting 'middle-tier-start'... [18:16:47] Finished 'middle-tier-start' after 1.06 ms [18:16:47] Starting 'selenium-start'... [18:16:47] using local Selenium server [18:16:48] Finished 'clean' after 399 ms [18:16:48] Starting 'bower-files'... [18:16:48] Finished 'bower-files' after 367 ms [18:16:48] Starting 'build'... Updating selenium standalone downloading https://selenium-release.storage.googleapis.com/2.45/selenium-server-standalone-2 Updating chromedriver downloading https://chromedriver.storage.googleapis.com/2.14/chromedriver_win32.zip... [18:16:49] processing files chromedriver_2.14.zip downloaded to C:\Users\builder\marklogic-samplestack\node_modules\protr [18:16:51] saw 221 files [18:16:52] Finished 'build' after 3.61 s selenium-server-standalone-2.45.0.jar downloaded to C:\Users\builder\marklogic-samplestack\no [18:16:54] Finished 'selenium-start' after 6.2 s Samplestack LDAP server up at: ldap://127.0.0.1:33388 Starting selenium standalone server... [launcher] Running 1 instances of WebDriver Selenium standalone server started at http://172.18.128.218:54107/wd/hub @landing-page Feature: Landing Page

Users who visit the "home page" of the application should see the explore state.

Scenario: The page title is correct # specs\features\as-anyone\landing-page.fe When I visit the "landing" page # specs\features\as-anyone\landing-page.fe Then the page title is "explore - samplestack" # specs\features\as-anyone\landing-page.fe

@explore-docs-by-resolved-only @explore Feature: Explore Docs By Resolved Only

When searching for documents, a facet filter may be applied to limit results to those documents which are only resolved.

Scenario: As a contributor filtering by resolved only, I see the correct results Given I am a contributor And I am using the brief seed data And I visit the "explore" page And I clear all filters When I filter documents by resolved only = "true" Then the docs count is "1903" When I focus on the "first" search result Then the result "title" is "Q: Mary's Question Number 0" When I focus on the "last" search result Then the result "title" is "Q: Show modal if page refreshed, but not if a link is followe

@explore-docs-by-date-range @explore Feature: Explore Docs By Date Range

When searching for documents, a facet filter may be applied to limit results to those documents which created by the user.

@explore-docs-by-mine-only @explore Feature: Explore Docs By Mine Only

When searching for documents, a facet filter may be applied to limit results to those documents which created by the user.

Scenario: As a contributor filtering by mine only, I see the correct results # specs\feat Given I am "Joe" # specs\feat When I visit the "ask" page # specs\feat And I type "mine only test" as the question title # specs\feat And I type "test" as the question content # specs\feat And I enter "e2eTests" as a question tag # specs\feat And I submit the question # specs\feat And I visit the "explore" page # specs\feat And I clear all filters # specs\feat And I clear the search text # specs\feat When I filter documents by mine only = "true" # specs\feat Then the docs count is less than "100" # specs\feat

@ask-question Feature: Ask Question

Any Contributor may ask questions with Markdown formatting, associating one or more (existing) "tags" with the question.

Scenario: Ask a question with Markdown preview # specs\features\as-mary\ask-question.f Given I am "Mary" # specs\features\as-mary\ask-question.f And my user name is "MaryAdmin" # specs\features\as-mary\ask-question.f When I visit the "ask" page # specs\features\as-mary\ask-question.f And I type "" as the question title # specs\features\as-mary\ask-question.f Then the submit button is disabled # specs\features\as-mary\ask-question.f When I type "foo" as the question title # specs\features\as-mary\ask-question.f And I type "bar" as the question content # specs\features\as-mary\ask-question.f And I enter "javascr" as a question tag # specs\features\as-mary\ask-question.f Then the question title is "foo" # specs\features\as-mary\ask-question.f And the question content is "bar" # specs\features\as-mary\ask-question.f And the question tags menu appears # specs\features\as-mary\ask-question.f And the submit button is not disabled # specs\features\as-mary\ask-question.f And the previewed content is not displayed # specs\features\as-mary\ask-question.f When I click a question tag in the menu # specs\features\as-mary\ask-question.f Then the question tag is "javascript" # specs\features\as-mary\ask-question.f When I preview the content # specs\features\as-mary\ask-question.f Then the previewed content is displayed # specs\features\as-mary\ask-question.f And the previewed content has "strong" formatting # specs\features\as-mary\ask-question.f And I submit the question # specs\features\as-mary\ask-question.f Then the page title is "doc - samplestack" # specs\features\as-mary\ask-question.f

@vote-on-question Feature: Vote on Question

Contributors may vote questions up or down. One may only vote on a question once. When an upvote is made to a quesiton, the author of the question gains reputation points. When a downvote is made to a question, the author of the question loses reputation points.

Scenario: Mary votes for Joe's question # specs\features\as- Given I am "Joe" # specs\features\as- When I visit the "ask" page # specs\features\as- And I type "test votes" as the question title # specs\features\as- And I type "test" as the question content # specs\features\as- And I enter "e2eTests" as a question tag # specs\features\as- And I submit the question # specs\features\as- And the question id is known as "qid" # specs\features\as- And I am "Mary" # specs\features\as- When I visit the "qnadoc" page with id equal to "qid" # specs\features\as- And I focus on the question # specs\features\as- And the content contributor reputation is known as "reputation" # specs\features\as- And I vote it up # specs\features\as- Then the content contributor reputation is greater than "reputation" # specs\features\as- AssertionError: expected 51 to be above 51

@list-docs Feature: List Docs

When no search criteria are specified, the explore page lists documents only according to any applied filters. Such a list is not considered a true "search". When listing documents, the results are displayed differently. In particular, snippets are not displayed and the default sort is different.

Scenario: As a visitor I can see some, but not all of the docs # specs\features\as-vi Given I am a visitor # specs\features\as-vi And I am using the brief seed data # specs\features\as-vi When I visit the "explore" page # specs\features\as-vi Then the docs count is "1903" # specs\features\as-vi

@explore-docs-by-text @explore Feature: Explore Docs By Text

When searching for documents, a text filter may be applied to limit results to those documents which match the search text criteria.

Scenario: As a visitor searching for "Math", I see the correct results Given I am a visitor And I am using the brief seed data When I visit the "explore" page And I perform a search for "Math" Then the docs count is "54". When I focus on the "first" search result, Then the result "title" is "Q: How can I detect the distance that the user's mouse has mo AssertionError: expected 'Q: Get element -moz-transform:rotate value in jQuery' to equa

When I focus on the "last" search result,
Then the result "title" is "Q: Nested For loop for initializing array of arrays for Javas

Scenario: As a contributor searching for "Math", I see the correct results Given I am a contributor And I am using the brief seed data When I visit the "explore" page And I perform a search for "Math" Then the docs count is "77". When I focus on the "first" search result, Then the result "title" is "Q: How can I detect the distance that the user's mouse has mo AssertionError: expected 'Q: Get element -moz-transform:rotate value in jQuery' to equa

When I focus on the "last" search result,
Then the result "title" is "Q: Length of Number in JavaScript"

@login Feature: Login

The application allows both authenticated and unauthenticated access through login and logout operations. Authenticated users have access to more content and features.

Scenario: Try to log in, fail, then succeed # specs\features\login-logout\login.feat Given I am a visitor # specs\features\login-logout\login.feat And I am using the brief seed data # specs\features\login-logout\login.feat When I start to log in # specs\features\login-logout\login.feat And I attempt to log in with invalid credentials # specs\features\login-logout\login.feat And my login attempt is denied # specs\features\login-logout\login.feat And I attempt to log in as a Contributor # specs\features\login-logout\login.feat Then I am logged in # specs\features\login-logout\login.feat

Scenario: Log In as Mary # specs\features\login-logout\login.feature:17 Given I am "Mary" # specs\features\login-logout\login.feature:18 Then my user name is "MaryAdmin" # specs\features\login-logout\login.feature:19

@view-contributor Feature: View Contibutor

Any user may view "contributor detail" about themselves or other contributors of content to the system. If the user is a "local contributor", a custom dialog displays basic information. If the user is a StackOverflow user, then a new window is opened to display that user's StackOverflow profile.

Scenario: View Contributor Mary # specs\feat Given I am "Mary" # specs\feat And I am using the brief seed data # specs\feat And I visit the "qnadoc" page with id "5dce8909-0972-4289-93cd-f2e8790a17fc" # specs\feat And I focus on the question # specs\feat And I view the content contributor # specs\feat Then the contributor display name is "MaryAdmin" # specs\feat And the contributor votes cast are greater than "4" # specs\feat And the contributor reputation is greater than "99" # specs\feat And dismiss the dialog # specs\feat

Scenario: View Contributor Joe # specs\feat Given I am "Joe" # specs\feat And I am using the brief seed data # specs\feat And I visit the "qnadoc" page with id "778d0b9c-419f-496a-a300-44815d79708d" # specs\feat And I focus on the question # specs\feat And I view the content contributor # specs\feat Then the contributor display name is "JoeUser" # specs\feat And the contributor votes cast are greater than "2" # specs\feat And the contributor reputation is greater than "49" # specs\feat And dismiss the dialog # specs\feat

(::) failed steps (::)

AssertionError: expected 51 to be above 51

AssertionError: expected 'Q: Get element -moz-transform:rotate value in jQuery' to equal 'Q:

AssertionError: expected 'Q: Get element -moz-transform:rotate value in jQuery' to equal 'Q:

Failing scenarios: C:\Users\builder\marklogic-samplestack\specs\features\as-mary\vote-on-question.feature:9 # Sc C:\Users\builder\marklogic-samplestack\specs\features\explore-docs-by-text.feature:7 # Scenar C:\Users\builder\marklogic-samplestack\specs\features\explore-docs-by-text.feature:18 # Scena

12 scenarios (3 failed, 9 passed) 105 steps (3 failed, 4 skipped, 98 passed) Shutting down selenium standalone server. [launcher] 0 instance(s) of WebDriver still running [launcher] chrome #1 failed 3 test(s) [launcher] overall: 3 failed spec(s) [launcher] Process exited with error code 1 Stopping LDAP worker...

[laurelnaiad]

This appears to be the same one recently verified to have been fixed...

[clockworked247]

AssertionError: expected 'Q: Get element -moz-transform:rotate value in jQuery' to equal 'Q: AssertionError: expected 'Q: Get element -moz-transform:rotate value in jQuery' to equal 'Q: has to do with the new algorithm changes that went in recently. So I am just updating the tests to match the expected change in search results.

I saw AssertionError: expected 51 to be above 51 on the first execution of gulp e2e, but in subsequent executions all the tests passed. I'm looking into why that is now.

[clockworked247]

Running the exact same steps manually for "Vote on Question" work fine. But I did notice that the first time I clicked vote, there was a slight delay before the UI changed. With subsequent votes this issue was not observed. Digging deeper to see if I can figure out why that might be.

[gghai]

For QA : Also verify (MAC)

gulp e2e --sauce=osx10.10-firefox-34

once the bug is fixed , as of now its giving out 3 failures too .

[clockworked247]

Yea the failures are platform & browser independent. @gghai While you're at it, can you verify a second run only returns two errors?

[clockworked247]

Ok, so my best guess is we have e2e timing problems on the first vote.

I've been looking over the step definition code and fixed a few differences with regard to the voting test, but none have impacted the outcome of the test. Every time you run that test after a clean setup of the DB, it fails the "contributor reputation is greater than" check. But only on the first run. Run it again, and it works.

I noticed in my manual tests that the first vote took 281ms where the subsequent votes took only 63ms. But the bit of lag on the first attempt shouldn't matter anyway… right @laurelnaiad ?

  this.Then(
    /content contributor reputation is greater than "(.*)"/,
    function (repAlias, next) {
      var self = this;

      expect(this.currentPage.focusedItem.metadata.reputation)
        .to.eventually.be.greaterThan(self[repAlias])
        .then(this.notifyOk(next), next);
    }

The ".to.eventually.be.greaterThan" should wait for the value to update, right?

If you have any other ideas on why this would fail only on first run after db install, I'm all ears. It seems like a timing problem, but I could be wrong. I'm still pretty n00b when it comes to protractor/chai.

[gghai]

@clockworked247 That is correct , i ran the above command twice , 1st time -> 3 failures , 2nd time -> 2 failures .

[clockworked247]

I've got those two fixed, it's just the spooky first run one having to do with reputation.

I think I'll just commit what I have for now and keep on this phantom test bug.

[clockworked247]

@gghai Fixes merged, I'm still trying to figure out the error on first run of e2e tests. It's good to know that it's not indicative of actual breakage, but just some problem with the tests.

[gghai]

I can confirm we are now down to last 1 failure.

[clockworked247]

and only on first run, yes?

[gghai]

Yes !!

[laurelnaiad]

the bit of lag on the first attempt shouldn't matter anyway… right @laurelnaiad ?

Right -- *unless something is timing out the test (but I doubt that).

The ".to.eventually.be.greaterThan" should wait for the value to update, right?

Technically what it does is resolve the returned value as a promise and evaluate the result of that.... if the promise is resolved before the value is updated, then the test would fail, for instance.

[clockworked247]

I threw in a couple log statements and am seeing the ".to.eventually.be.greaterThan" statement finish even before the vote route is hit.

Failure execution order:

And I vote it up (test output) * START contributor reputation is greater than * * greaterThan checked FAILURE * * vote route start * * vote complete *

Successful execution:

* vote route start * And I vote it up (test output) * START contributor reputation is greater than * * vote complete * * greaterThan checked SUCCESS *

So in the failure case, the request to the middle tier only happens after the eventually statement finishes. What's confusing is this only occurs on first run... I'm pretty stumped on why there is a difference at all.

[laurelnaiad]

It could be a bug in the protractor promises, the selenium promises, or the Samplestack promises, or the Chai-as-promised promises, or... :smile:

[clockworked247]

Yea, taking lots of stabs in the dark to try to figure it out where the breakage is. Nothing has hit the mark yet.

[laurelnaiad]

This is actually a dupe of #564