search

rubyforgood / playtime

Supporting children and families experiencing homelessness in Washington, DC. Live app - https://wishlist.playtimeproject.org Organization Website:
http://www.playtimeproject.org/
MIT License
89 stars 50 forks source link

Investigate random Travis failures #102

Closed leesharma closed 6 years ago

leesharma commented 7 years ago

It seems like we've had an unusual number of issues with Travis failing/passing on restart. Most of the failures seem to be feature tests failing in chunks because Capybara can't find text/selectors. Re-running the build usually works.

Is there anything we can do about this?

willmurphyscode commented 7 years ago

Is it possible to add some links to failed build logs or to the specific failing Capybara tests to this issue? I'd like to help but I haven't been able to find anything.

leesharma commented 7 years ago

I don't have any handy unfortunately! Next time I see them, I'll be sure to copy the output. The errors are intermittent, and running the same seed doesn't reproduce them.

I think you were probably right when we spoke the other day–it sounds like something's timing out.

seanmarcia commented 7 years ago

I also dug into this a while ago and couldn't reproduce it :(

leesharma commented 7 years ago

Okay, I ran the tests a couple dozen times on master and reproduced it locally. Here is a gist with test output.

In this example, two feature test files are failing: managing_users_spec and pledging_an_item_spec. Usually all the specs in a feature test file will fail, though the actual files that fail vary.

It might also be tied to system resources. I have an ultrabook, and unscientifically, specs seem to fail more often when I'm streaming a video or have an IDE open.

seanmarcia commented 7 years ago

As an FYI, it doesn't appear to be an rspec test ordering issue as I ran the tests using the same seed that failed for lee and they worked for me.

leesharma commented 7 years ago

@seanmarcia finally broke the travis build! It only has one failure this time though, which is odd–I guess it's not always the whole file:

Randomized with seed 12215

...............................................................................F......................................................................................................................................................................................................................................................................

Failures:

  1) Managing Users: As an admin I can delete a user account

     Failure/Error: expect(page).to have_text "User was successfully destroyed."

       expected to find text "User was successfully destroyed." in "Home(current) Wishlists Admin Tools Users Pledges New Wishlist Hello, admin@example.com Log Out × This account is the only remaining Admin user. Please assign another Admin before deleting this account Export CSV Listing Users Name Email Admin user74@example.com Show Edit Destroy user72@example.com Show Edit Destroy user73@example.com Show Edit Destroy admin@example.com Show Edit Destroy"

     # ./spec/features/managing_users_spec.rb:88:in `block (3 levels) in <top (required)>'

Finished in 36.04 seconds (files took 4.76 seconds to load)

342 examples, 1 failure

Failed examples:

rspec ./spec/features/managing_users_spec.rb:79 # Managing Users: As an admin I can delete a user account
seanmarcia commented 7 years ago

Finally got it to break on travis, here is travis config and log: https://gist.github.com/seanmarcia/a9306f0b5bb1a8c039e44b5451c2309a

Link to travis job: https://travis-ci.org/rubyforgood/playtime/jobs/287380754