search

AgileVentures / WebsiteOne

A website for Agile Ventures
https://www.agileventures.org
Other
143 stars 248 forks source link

intermittent js failure? #806

Closed tansaku closed 8 years ago

tansaku commented 8 years ago 
 Scenario: Insert media model rejects badly formatted youtube links                                      # features/documents.feature:170
    Given I am logged in                                                                                  # features/step_definitions/user_steps.rb:19
    And I am using the Mercury Editor to edit document "Guides"                                           # features/step_definitions/mercury_steps.rb:27
    And I am focused on the "document body" within the Mercury Editor                                     # features/step_definitions/contained_search_steps.rb:33
    And I click on the "Insert Media" button within the Mercury Toolbar                                   # features/step_definitions/contained_search_steps.rb:33
    And I fill in "YouTube URL" with "https://www.youtube.io/watch?v=foo" within the Mercury Editor Modal # features/step_definitions/contained_search_steps.rb:33
    And I click "Insert Media" within the Mercury Editor Modal                                            # features/step_definitions/contained_search_steps.rb:33
    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)× Images URL Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:167:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:177:in `Then I should see "is invalid" within the Mercury Modal'
    And the Mercury Editor modal window should be visible                                                 # features/step_definitions/mercury_steps.rb:61

Failing Scenarios: cucumber features/documents.feature:170 # Scenario: Insert media model rejects badly formatted youtube links

tansaku commented 8 years ago

got this for a second time today on a trivial documentation update

tansaku commented 8 years ago

and a third:

Failing Scenarios:
cucumber features/documents.feature:159 # Scenario: Insert media model accepts full url youtube links
cucumber features/documents.feature:170 # Scenario: Insert media model rejects badly formatted youtube links
tansaku commented 8 years ago

and a forth: 

cucumber features/documents.feature:159
freeranger commented 8 years ago

I get this quite frequently on my ci build of WSO which is basically a clone of the main one - it can happen two or three times in a row and then just work.....possibly a timing issue somewhere?

tansaku commented 8 years ago

most likely

freeranger commented 8 years ago

Could these two scenarios be disabled until someone looks at the issue? I get one or both failing pretty constantly on my semaphore build - it's only pot luck if it works or not. The downside ofc to disabling is that nobody looks at them because the issue has "gone away"...

tansaku commented 8 years ago

yes, we need to get to the bottom of these - @decareano posted a great link on the subject in localsupport - if I can stabilize the grader I might try to make this my priority tomorrow:

https://spin.atomicobject.com/2012/04/27/intermittent-test-failures/

mtc2013 commented 8 years ago

This intermittently failed for me:-

Failures:

1) EventsController GET show assigns a event instance Failure/Error: expect(assigns(:recent_hangout)).to eq(event_instance)

   expected: #<EventInstance id: 43, event_id: 23, title: "Hangout_91", hangout_url: "http://hangout.test", created_at: "2016-03-02 14:16:09", updated_at: "2016-03-02 14:16:09", uid: "uid_102", category: "Category_89", project_id: 41, user_id: 194, yt_video_id: "yt_video_id_84", participants: [["91", {:person=>{:displayName=>"Broadcaster91", :id=>"youtube_id_91", :isBroadcaster=>"true"}}], ["91", {:person=>{:displayName=>"Participant_91", :id=>"youtube_id_91", :isBroadcaster=>"false"}}]], hoa_status: nil>
        got: nil

   (compared using ==)
 # ./spec/controllers/events_controller_spec.rb:32:in `block (3 levels) in <top (required)>'
tansaku commented 8 years ago

got another random failure

(::) failed steps (::)

expected to find css "img[src*=\"/assets/mercury/missing-image.png\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/basic_steps.rb:348:in `/^I should see an image with source "([^"]*)"$/'
./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
features/documents.feature:155:in `Then I should see an image with source "/assets/mercury/missing-image.png" within the Mercury Editor'

Failing Scenarios:
cucumber features/documents.feature:146 # Scenario: Missing Image gets added when a user can inserts an invalid image
tansaku commented 8 years ago

I just got the same rspec failure as @mtc2013 had locally

  1) EventsController GET show assigns a event instance
     Failure/Error: expect(assigns(:recent_hangout)).to eq(event_instance)

       expected: #<EventInstance id: 100, event_id: 92, title: "Hangout_210", hangout_url: "http://hangout.test", created_at: "2016-03-03 08:05:58", updated_at: "2016-03-03 08:05:58", uid: "uid_224", category: "Category_186", project_id: 211, user_id: 587, yt_video_id: "yt_video_id_209", participants: [["216", {:person=>{:displayName=>"Broadcaster216", :id=>"youtube_id_216", :isBroadcaster=>"true"}}], ["216", {:person=>{:displayName=>"Participant_216", :id=>"youtube_id_216", :isBroadcaster=>"false"}}]], hoa_status: nil>
            got: nil

       (compared using ==)
     # ./spec/controllers/events_controller_spec.rb:32:in `block (3 levels) in <top (required)>'

went away on a second run ...

tansaku commented 8 years ago

just got three docs failures in CI again:

cucumber features/documents.feature:146 # Scenario: Missing Image gets added when a user can inserts an invalid image
cucumber features/documents.feature:159 # Scenario: Insert media model accepts full url youtube links
cucumber features/documents.feature:170 # Scenario: Insert media model rejects badly formatted youtube links
tansaku commented 8 years ago

this in @freerangers cucumber pull request: 

  Scenario: Show a document                                      # features/documents.feature:47
    Given I am on the "Show" page for project "hello mars"       # features/step_definitions/basic_steps.rb:256
    When I click the sidebar link "Guides"                       # features/step_definitions/documents_steps.rb:44
    Then I should be on the "Show" page for document "Guides"    # features/step_definitions/basic_steps.rb:252
    And I should see "Guides"                                    # features/step_definitions/basic_steps.rb:185
    And I should see "New content 0"                             # features/step_definitions/basic_steps.rb:185
    And I should see a link to "Edit" page for document "Guides" # features/step_definitions/basic_steps.rb:260
      expected to find link or button nil with text "Guides" but there were no matches. Also found "", "Log in", "Sign up", "", "", "About", "Projects", "Members", "Articles", "Events", "Upcoming events", "Past scrums", "Past events", "Getting started", "", "", "hello world", "Documentation", "", "hello mars", "", "", "", "Become a supporter", "×", "List of Projects", "hello mars", "Revisions", "comments powered by Disqus.", "comments powered by Disqus", "About Us", "Getting Started", "Dashboard", "Facebook", "Twitter", "Makers Academy", "AirPair", "Becoming a sponsor", "info@agileventures.org", which matched the selector but not all filters. (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:263:in `/^I should( not be able to)? see a link to "([^"]*)" page for ([^"]*) "([^"]*)"$/'
      features/documents.feature:54:in `And I should see a link to "Edit" page for document "Guides"'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-03-12-33-13.782.html
  @javascript
  Scenario: Insert media model accepts full url youtube links                                                            # features/documents.feature:159
    Given I am logged in                                                                                                 # features/step_definitions/user_steps.rb:19
    And I am using the Mercury Editor to edit document "Guides"                                                          # features/step_definitions/mercury_steps.rb:27
    And I am focused on the "document body" within the Mercury Editor                                                    # features/step_definitions/contained_search_steps.rb:33
    And I click on the "Insert Media" button within the Mercury Toolbar                                                  # features/step_definitions/contained_search_steps.rb:33
    And I fill in "YouTube URL" with "https://www.youtube.com/watch?v=foo" within the Mercury Editor Modal               # features/step_definitions/contained_search_steps.rb:33
    And I click "Insert Media" within the Mercury Editor Modal                                                           # features/step_definitions/contained_search_steps.rb:33
    Then the Mercury Editor modal window should not be visible                                                           # features/step_definitions/mercury_steps.rb:61
    And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:352:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:167:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-03-12-33-37.869.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-03-12-33-37.869.png
Failing Scenarios:
cucumber features/documents.feature:47 # Scenario: Show a document
cucumber features/documents.feature:159 # Scenario: Insert media model accepts full url youtube links
tansaku commented 8 years ago

so cucumber features/documents.feature:47 was consistently failing for @freeranger while cucumber features/documents.feature:159 is one of our intermittent failures ... highlighting how confusing intermittent failures can be

mtc2013 commented 8 years ago 
@mercury_step @javascript
  Scenario: Editing Pitch content with Mercury Editor                                                    # features/projects.feature:184
    Given I am logged in                                                                                 # features/step_definitions/user_steps.rb:19
    And I am on the "Show" page for project "hello mars"                                                 # features/step_definitions/basic_steps.rb:256
    And I click the "Join Project" button                                                                # features/step_definitions/basic_steps.rb:77
    And I am using the Mercury Editor to edit project "hello mars"                                       # features/step_definitions/mercury_steps.rb:27
    When I fill in the editable field "Pitch" for "project" with "This is my exciting marketing content" # features/step_definitions/mercury_steps.rb:9
    And I click "Save" within Mercury Editor toolbar                                                     # features/step_definitions/mercury_steps.rb:1
    Then I should see a flash "The project has been successfully updated."                               # features/step_definitions/basic_steps.rb:193
      expected to find css ".alert" with text "The project has been successfully updated." but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:195:in `/^I should( not)? see a flash "([^"]*)"$/'
      features/projects.feature:191:in `Then I should see a flash "The project has been successfully updated."'
    Then I should be on the "Show" page for project "hello mars"                                         # features/step_definitions/basic_steps.rb:252
    And I should see "This is my exciting marketing content"                                             # features/step_definitions/basic_steps.rb:185
    HTML screenshot: /Users/michael/Desktop/WebsiteOne/tmp/capybara/screenshot_2016-03-08-14-28-34.642.html
    Image screenshot: /Users/michael/Desktop/WebsiteOne/tmp/capybara/screenshot_2016-03-08-14-28-34.642.png
freeranger commented 8 years ago

Is it me or is it getting worse? I pretty much can't get any semaphore build green now....

tansaku commented 8 years ago

it feels like something is going on ...

tansaku commented 8 years ago

please post specifics of fails that you get here @freeranger - it helps us debug

freeranger commented 8 years ago

Some examples:

Scenario: See my avatar on My account page                                                                      # features/users/avatar.feature:10
    When I click pulldown link "My account"                                                                       # features/step_definitions/user_steps.rb:259
      Ambiguous match, found 2 elements matching link or button "My account" (Capybara::Ambiguous)
      ./features/step_definitions/user_steps.rb:261:in `/^I click pulldown link "([^"]*)"$/'
      features/users/avatar.feature:11:in `When I click pulldown link "My account"'
    Then I should see my avatar image                                                                             # features/step_definitions/avatar_steps.rb:1
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-15-21-25-55.123.html
@javascript
  Scenario: Insert media model accepts full url youtube links                                                            # features/documents.feature:157
    Given I am logged in                                                                                                 # features/step_definitions/user_steps.rb:19
    And I am using the Mercury Editor to edit document "Guides"                                                          # features/step_definitions/mercury_steps.rb:27
    And I am focused on the "document body" within the Mercury Editor                                                    # features/step_definitions/contained_search_steps.rb:33
    And I click on the "Insert Media" button within the Mercury Toolbar                                                  # features/step_definitions/contained_search_steps.rb:33
    And I fill in "YouTube URL" with "https://www.youtube.com/watch?v=foo" within the Mercury Editor Modal               # features/step_definitions/contained_search_steps.rb:33
    And I click "Insert Media" within the Mercury Editor Modal                                                           # features/step_definitions/contained_search_steps.rb:33
    Then the Mercury Editor modal window should not be visible                                                           # features/step_definitions/mercury_steps.rb:61
    And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:356:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:165:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-16-09-23-59.288.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-16-09-23-59.288.png
@javascript
  Scenario: Closing an existing video and opening a new one should update the player # features/scrums.feature:19
    Given I visit "/scrums"                                                          # features/step_definitions/basic_steps.rb:60
    And I click the first scrum in the timeline                                      # features/step_definitions/scrums_steps.rb:24
    Then I should see a modal window with the first scrum                            # features/step_definitions/scrums_steps.rb:13
    When I close the modal                                                           # features/step_definitions/scrums_steps.rb:36
    And I click the second scrum in the timeline                                     # features/step_definitions/scrums_steps.rb:30
    Then I should see a modal window with the second scrum                           # features/step_definitions/scrums_steps.rb:13
      expected to find css "#playerTitle" but there were no matches. Also found "", which matched the selector but not all filters. (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/scrums_steps.rb:17:in `/^I should see a modal window with the (first|second) scrum$/'
      features/scrums.feature:25:in `Then I should see a modal window with the second scrum'
capybara-screenshot could not detect a screenshot driver for 'poltergeist_debug'. Saving with default with unknown results.
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-01-04.211.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-01-04.211.png
tansaku commented 8 years ago

thanks @freeranger - I've been trying to mark the intermittent failure points in this branch: https://github.com/AgileVentures/WebsiteOne/tree/806_failure_points which also uses some cucumber hooks to try and give those scenarios longer to run.

I suspect that we won't be able to achieve stability without losing some functionality or adding proper sandboxing via puffing_billy or vcr or possibly both

freeranger commented 8 years ago

hmm...may be beyond my current skill level so no promises but I'll see what I can do....

freeranger commented 8 years ago

The Mercury Editor tests seem failing before it even gets to post to youtube so puffing_billy won't help here. This is the sort of thing I see for example when the "Insert media model accepts full url youtube links" scenario fails: image Where it failed to fill in the YouTube URL field so there is nothing on screen when submit is pressed so validation is failing.

freeranger commented 8 years ago

I increased the timeout and did a load of builds (21 at last count) here:

https://semaphoreci.com/freeranger/websiteone/branches/806_intermittent_failures Build Status

And a lot of them worked, more than some of the other builds of different branches that I kicked off around the same time - though I did get one new failure:

image

As I said yesterday, I think it is largely, if not entirely, down to not entering the YouTuble URL video so validation fails and the page is never submitted. I tried various things to select the radio button next to it first (rather than start typing in the text field, which results in the radio being selected) but I couldn't find the radio on the page - I don't know if it is the complexity of having this as an iframe or just my lack of capybara skills, but trying various suggestions from mr google - xpath selections, css finds etc, didn't work for me....and perhaps wouldn't have made any difference anyway...

It's not a timeout issue and it's not a network (as in talking to youtube etc) issue - just seems to be intermittently failing to fill in forms. Also, IME, seems to happen much more on semaphore than it does running locally...

tansaku commented 8 years ago

Thanks for this review.

I think puffing billy might still help because we are not "posting to youtube", we are posting to our own server. What puffing billy sandboxes is not a post to our own server (which we complete in these acceptance tests), but all the network connections going out from the browser to load remote javascripts and images etc. All things that can block and slow things down and create intermittent failures when javascript is trying to get the right forms on the screen.

You might be right that it won't help in this case, but we do need to add it, and we should try it.

freeranger commented 8 years ago

@tansaku If you look at the screenshot from the weekend though, the form is indeed loaded The problem is there is nothing filled in on the form because this And I fill in "YouTube URL" with "https://www.youtube.com/watch?v=foo" within the Mercury Editor Modal isn't actually filling in the form. Even if some remote javascript hadn't loaded, would that affect capybara finding the control and populating it? capybara itself is already loaded and has been able to bring up the form by clicking on a link or button....

tansaku commented 8 years ago

It's a wild card so its something we should lock down. I find that part of the trick to making progress with this kind of error is to not be a stickler for the logic of it, but make sure the hatches are battened down.

Also you don't know that the screenshot is completely accurate. Anyway, there comes a point that if you don't have a specific alternative to try, then why not "just try" the thing that I am suggesting, even if at the moment you can't see the logic of it ...? :-)

We can argue back and forth the logic which takes up a lot of time - more time than actually just trying it perhaps ...

freeranger commented 8 years ago

Fair enough. I would like to have tried selecting the radio button rather than relying on the focus of the text box magically selecting it, but I couldn't get that working. setting up puffing billy is beyond me at the moment - I've only barely scratched the surface of WSO to understand it, nevermind this added complexity - back to you or some other volunteer I'm afraid....

tansaku commented 8 years ago

fair enough - see #932

tansaku commented 8 years ago

More failures:

cucumber -p semaphoreci features/documents.feature:157 # Scenario: Insert media model accepts full url youtube links
cucumber -p semaphoreci features/documents.feature:168 # Scenario: Insert media model rejects badly formatted youtube links
cucumber -p semaphoreci features/scrums.feature:19 # Scenario: Closing an existing video and opening a new one should update the player

@yggie I'm trying to keep a record of all the places we see intermittent failures in this issue

    Then I should see a modal window with the second scrum                           # features/step_definitions/scrums_steps.rb:13
      expected "display: none; " to include "display: block;" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/scrums_steps.rb:15:in `/^I should see a modal window with the (first|second) scrum$/'
      features/scrums.feature:25:in `Then I should see a modal window with the second scrum'
    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)�� Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:175:in `Then I should see "is invalid" within the Mercury Modal'
    And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:360:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:165:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'
tansaku commented 8 years ago

and annoyingly our new stripe test has an intermittent failure too!

cucumber -p semaphoreci features/charge_activity.feature:6
Your userAgent seems to be not supported by inline-style-prefixer. Feel free to open an issue.
    And I fill in appropriate card details     # features/step_definitions/charge_steps.rb:1
      Unable to find field "cc-exp" (Capybara::ElementNotFound)
      ./features/step_definitions/charge_steps.rb:7:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/charge_steps.rb:3:in `/^I fill in appropriate card details$/'
      features/charge_activity.feature:9:in `And I fill in appropriate card details'
tansaku commented 8 years ago

and this one again:

    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)�� Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:175:in `Then I should see "is invalid" within the Mercury Modal'
cucumber -p semaphoreci features/documents.feature:168 # Scenario: Insert media model rejects badly formatted youtube links
tansaku commented 8 years ago

and another:

    Then I should see a modal window with the second scrum                           # features/step_definitions/scrums_steps.rb:13
      expected "display: none; " to include "display: block;" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/scrums_steps.rb:15:in `/^I should see a modal window with the (first|second) scrum$/'
      features/scrums.feature:25:in `Then I should see a modal window with the second scrum'
cucumber -p semaphoreci features/scrums.feature:19
tansaku commented 8 years ago

this looks interesting: https://bibwild.wordpress.com/2016/02/18/struggling-towards-reliable-capybara-javascript-testing/

tansaku commented 8 years ago

So I'm focusing on this last individual failure features/scrums.feature:19 and I note that if I reduce the timeout settings to 0 or 0.1 I get these kinds of errors:

Scenario: Closing an existing video and opening a new one should update the player # features/scrums.feature:19
    Given I visit "/scrums"                                                          # features/step_definitions/basic_steps.rb:64
    And I click the first scrum in the timeline                                      # features/step_definitions/scrums_steps.rb:24
      Timed out waiting for response to {"id":"d9ae4d24-8368-4982-9eaa-a8cee66f8880","name":"body","args":[]}. It's possible that this happened because something took a very long time (for example a page load was slow). If so, setting the Poltergeist :timeout option to a higher value will help (see the docs for details). If increasing the timeout does not help, this is probably a bug in Poltergeist - please report it to the issue tracker. (Capybara::Poltergeist::TimeoutError)
      ./features/step_definitions/scrums_steps.rb:25:in `/^I click the first scrum in the timeline$/'
      features/scrums.feature:21:in `And I click the first scrum in the timeline'

which are different in kind from the errors we get in batch. With a poltergeist timeout of 1 and a Capybara.default_max_wait_time of 1 the tests passes for me reliably on my laptop, when I have an internet connection.

I note however that despite our attempt to plug the billy cache leaks that just from running this single test I have a set of new cache entries:

    features/req_cache/get_s.ytimg.com_915edb4b5f224648b2607dc57119661559439dc1.yml
    features/req_cache/get_s.ytimg.com_97ea91ae8364c347e1bafbcf3a60d3830420c3e5.yml
    features/req_cache/get_www.google.com_dda3078e234d52b37f250121ebb31829d67f6628.yml
    features/req_cache/get_www.youtube.com_754eb41a2f135e6eafbc40d9a4acf987250d14d4.yml
    features/req_cache/get_www.youtube.com_771e045e849dcf0e8325651601d2397edca211c9.yml
    features/req_cache/get_www.youtube.com_7ce01254dc22fb0ec070a5d431931551248b7971.yml

indicating that this test is still hitting some remote servers ... none of which have signficant variation in their parameters ... but I note that several of them have (particularly the www.youtube.com ones) are getting 301 (moved permanently) responses and as a consequence may not be being cached effectively - I thought that puffing billy cached these by default, but there is an extra setting for that that I can add:

    non_successful_cache_disabled = false

I also note that if I wrap this test in a @vcr tag we also get server network connections being cached.

tansaku commented 8 years ago

hmm, interesting to note that if I put @vcr tags on the documents.feature (one of the prime culprits) I get a fresh load of billy leaks ...

    features/req_cache/get_a.disquscdn.com_005fa385265710de9b800462d9db61ef21ec4c99.yml
    features/req_cache/get_a.disquscdn.com_2aa0b18e4bd9e580dc09cf4f417b21146ddf05b6.yml
    features/req_cache/get_a.disquscdn.com_2bc23452ce0ff2a22e561bb4519a23945f0ba922.yml
    features/req_cache/get_a.disquscdn.com_bca73df499bedab281bd05c54b027f262189d85d.yml
    features/req_cache/get_a.disquscdn.com_d008b8a1c5d90ecb9fc1701353d06b9481fa230b.yml
    features/req_cache/get_disqus.com_442dc930b209a7b62857b10a08ab762f55c39f56.yml
    features/req_cache/get_google.com_8a28b678b788b1c051c987cd0d6f8c697a48b352.yml
    features/req_cache/get_s.ytimg.com_915edb4b5f224648b2607dc57119661559439dc1.yml
    features/req_cache/get_s.ytimg.com_97ea91ae8364c347e1bafbcf3a60d3830420c3e5.yml
    features/req_cache/get_ssl.google-analytics.com_7b9a51d3b1e9af83bfc375f4813db28046b99513.yml
    features/req_cache/get_www.google.com_dda3078e234d52b37f250121ebb31829d67f6628.yml
    features/req_cache/get_www.youtube.com_36a2809495d3decd3d207abd85e5162a07e86986.yml
    features/req_cache/get_www.youtube.com_e6558ad62c301081754b9ab9f6a15aeb1284f981.yml
tansaku commented 8 years ago

what's really odd is that these billy leaks don't appear when running in batch (without vcr) - need to check variations ...

tansaku commented 8 years ago

interesting - just running the document.feature by itself (without any vcr mods) generates new billy cache files, implying that somehow there is some different behaviour in terms of what sites the browser is hitting when we run the tests in batch and we run them at the feature file level.

For scrums I also saw this behaviour at the scenario level ...

tansaku commented 8 years ago

this implies some fundamental inconsistency in the browser based behaviour depending on whether we are in batch mode or individual feature or scenario mode, which is not a good thing ... locking down everything with vcr and billy will help reduce variation, but if there is something else causing this change between the different running modes then we won't get stability until we can identify and then either patch or remove that underlying problem ...

tansaku commented 8 years ago

Options:

1) comprehensive VCR sandboxing is required 2) upgrading all the relevant gems, e.g. capybara, phantomjs, poltergeist etc. 3) review features (and cucumber env settings one by one) --> start with a branch that had a single feature test - and check that passes on CI and developer machines, and then we would add in each acceptance in turn, checking what the tags mean - ensuring we have billy and vcr sandboxing, and working through everything to ensure maximum consistency ... could start with usual suspects 4) making tests run in fixed order (i.e. not randomised by cucumber) 5) play with rails concurrency settings 6) prune back on javascript tests - can some be thrown out? 7) comment out the intermittent tests - or just have them not run on CI using a tag like @intermittent_fail

freeranger commented 8 years ago

At least some of the common failures are when trying to select form elements and no network traffic should be involved at that point, as we have discussed earlier in the thread. It may be that the focus is in the wrong area since adding billy and/or vcr in those areas didn't fix the problems...so maybe it's more to do with field selection/validation than network reliability? Sandboxing may be a good idea generally, but not necessarily as a solution to these failures...

tansaku commented 8 years ago

thanks @freeranger - it may well be that network sandboxing will not fix this, however note that previously I only added billy, not VCR. What we have established so far is that billy by itself has not conclusively fixed these issues. Network variability causing other tests to hang around and influence each other may still be a strong component of this problem, since VCR has not yet been added throughout for a complete sandbox.

That's what we're doing in the cleanroom branch, as well as being on the lookout for other issues, but being able to do with a more manageable subset of the tests ... There seem to be several factors in play, and to be able to manage that I think we have to work hard to reduce all sources of external variation from our tests.

tansaku commented 8 years ago

so got this with what we think is fully sandboxed code:

    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)× Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:174:in `Then I should see "is invalid" within the Mercury Modal'
cucumber -p semaphoreci features/documents.feature:167 # Scenario: Insert media model rejects badly formatted youtube links
freeranger commented 8 years ago

Ah that's annoying, perhaps leading to where I was going with it (ie not the lack of sandboxing being the issue) but still, lot of good work there so even if it doesn't help at all for this case, the tests should be more reliable overall I should think - so don't be disheartened! Does this happen for you locally or was it a semaphore build that failed? Do you want people to blast it again a load of times to see if we get it? Have you tried it in an environment that wasn't a mac, maybe c9.io or an ubunto vm to see if you can get it to fail on a more regular basis and hopefully be easier to track down then?

tansaku commented 8 years ago

Another failure in a new location (this time in the old code):

 And I create a document named "A New Guide to the Galaxy" # features/step_definitions/activity_feed_steps.rb:15
    And I create a project named "Build NCC-1701 Enterprise"  # features/step_definitions/activity_feed_steps.rb:25
      Timed out waiting for response to {"id":"2866f7e2-cdae-49c3-b35e-05b0eaad0654","name":"click","args":[11,6]}. It's possible that this happened because something took a very long time (for example a page load was slow). If so, setting the Poltergeist :timeout option to a higher value will help (see the docs for details). If increasing the timeout does not help, this is probably a bug in Poltergeist - please report it to the issue tracker. (Capybara::Poltergeist::TimeoutError)
      ./features/step_definitions/basic_steps.rb:82:in `/^I click the "([^"]*)" button$/'
      ./features/step_definitions/activity_feed_steps.rb:26:in `/^I create a project named "([^"]*)"$/'
      features/public_activity.feature:23:in `And I create a project named "Build NCC-1701 Enterprise"'
capybara-screenshot could not detect a screenshot driver for 'poltergeist_billy_debug'. Saving with default with unknown results.
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-00-50.831.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-00-50.831.png

Failing Scenarios:
cucumber -p semaphoreci features/public_activity.feature:25

@freeranger to answer some of your questions:

1) that was a failure on CI 2) the latest (and now complete) cleanroom branch could be blasted a few times to see if we get local failures - @mtc2013 at least had none locally while we were developing the whole thing which involved about 40+ runs, and I only had one failure, but that was with some extra cache files that I then added ... 3) we haven't tried on other platforms apart from it being run on @diraulo, my and @mtc2013's systems. A repeat run on C9 is a good idea - particularly in that all our projects should have C9 setup guides - I tried to reduce timeouts to make more reliable failures, but made no progress there.

My main plan of attack next is once we have the cleanroom in develop is wait for the next failure and then pounce on it for a detailed analysis with all networking variation removed ... there's also making the cleanroom work with ruby 2.3 ...

freeranger commented 8 years ago

@tansaku Ok I'll try to get a c9 build up and running at the weekend and give it a good few runs.

tansaku commented 8 years ago

just got this locally:

expected to find css ".alert" with text "The project has been successfully updated." but there were no matches (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/basic_steps.rb:199:in `/^I should( not)? see a flash "([^"]*)"$/'
features/projects.feature:192:in `Then I should see a flash "The project has been successfully updated."'

Failing Scenarios:
cucumber features/projects.feature:185 # Scenario: Editing Pitch content with Mercury Editor

but also had some stray network connections:

:scope: 
:url: http://csi.gstatic.com/csi?v=3&s=gwidget&action=hangout&it=wdi.116,wri.116&srt=21&e=abc_e81,abc_e97,abc_e100,abc_e122,abc_e123,abc_e30,abc_e79,abc_e127,abc_l0,abc_m0,abc_n0,abc_m0n0,abc_u0&rt=
:body: ''
:status: 204
:method: get
:headers:
  Access-Control-Allow-Origin: "*"
  Date: Thu, 14 Apr 2016 14:45:53 GMT
  Pragma: no-cache
  Expires: Fri, 01 Jan 1990 00:00:00 GMT
  Cache-Control: no-cache, no-store, must-revalidate
  Last-Modified: Wed, 21 Jan 2004 19:51:30 GMT
  Content-Type: image/gif
  Server: Golfe2
  Content-Length: '0'
  Connection: close
:content: ''
tansaku commented 8 years ago

another CI failure:

 And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:360:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:164:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-04-14-15-14-41.579.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-04-14-15-14-41.579.png
Failing Scenarios:
cucumber -p semaphoreci features/documents.feature:156 # Scenario: Insert media model accepts full url youtube links
tansaku commented 8 years ago

So we've got several more intermittent fails on CI with the new setup:

features/documents.feature:156
features/documents.feature:167
features/scrums.feature:22

The new sandboxed setup seems (?) more stable on individual developer machines, but if anything these fails on CI seem even more common? At least if something is reliably failing we have a better chance of fixing it ...

tansaku commented 8 years ago

I was taking a little look at features/documents.feature:167 in more detail locally (it passes single and in batch on my local machine).

It occurred to me that perhaps there is a way to reset the JS browser inbetween test runs - which led me to find this post:

https://github.com/teampoltergeist/poltergeist/issues/375

which gives us another thing to experiment with. What I tried first was printing out images around the elements that fails on CI like so:

  @javascript
  Scenario: Insert media model rejects badly formatted youtube links
    Given I am logged in
    And I am using the Mercury Editor to edit document "Guides"
    And I am focused on the "document body" within the Mercury Editor
    And I click on the "Insert Media" button within the Mercury Toolbar
    And I fill in "YouTube URL" with "https://www.youtube.io/watch?v=foo" within the Mercury Editor Modal
    And save a screenshot of the page at "/tmp/before_insert.png"
    And I click "Insert Media" within the Mercury Editor Modal
    And save a screenshot of the page at "/tmp/after_insert.png"
    Then I should see "is invalid" within the Mercury Modal
    And the Mercury Editor modal window should be visible

now interestingly that fails for me locally the way it fails on CI and the screen shots look like this:

before insert

after insert

tansaku commented 8 years ago

and the error I get is this:

  And I click "Insert Media" within the Mercury Editor Modal                                            # features/step_definitions/contained_search_steps.rb:33
    And save a screenshot of the page at "/tmp/after_insert.png"                                          # features/step_definitions/basic_steps.rb:275
    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)× Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:176:in `Then I should see "is invalid" within the Mercury Modal'
    And the Mercury Editor modal window should be visible                                                 # features/step_definitions/mercury_steps.rb:61
    HTML screenshot: /Users/tansaku/Documents/GitHub/AgileVentures/WebsiteOne/tmp/capybara/screenshot_2016-04-15-12-08-29.698.html
    Image screenshot: /Users/tansaku/Documents/GitHub/AgileVentures/WebsiteOne/tmp/capybara/screenshot_2016-04-15-12-08-29.698.png

and the dumped screen looks like this:

the same stuff that @freeranger was looking at previously ...

tansaku commented 8 years ago

and this is kind of strange, since it makes it look like the youtube URL has been inserted correctly with the correct radiobutton clicked, but then after insert media is clicked the modal reloads with a different kind of issue? Has the invalid issue appeared and then disappeared? Are we running too fast or too slow here ... weird ...