search

hotsh / rstat.us

Simple microblogging network based on the ostatus protocol.
http://rstat.us/
Other
722 stars 215 forks source link

Figure out the 4+ tests that sporadically fail #479

Open carols10cents opened 12 years ago

carols10cents commented 12 years ago

We have one skipped test that sometimes fails on travis but doesn't seem to fail locally, and there's another one that is NOT currently skipped that has the same behavior. It's this one:

test_0002_unfollows_another_user(FollowingSpec::OnRstatUsSpec): Capybara::ElementNotFound: no button with value or id or text 'unfollow-4f2c40089870160d4d000947' found

See these failing builds to see the test(s?) I'm talking about:

http://travis-ci.org/#!/hotsh/rstat.us/builds/625838 http://travis-ci.org/#!/hotsh/rstat.us/builds/622687 http://travis-ci.org/#!/hotsh/rstat.us/builds/622614

There's pretty much no way the changes in those builds could have caused those tests to fail, so there's something else going on here.

It's really getting annoying, so if anyone figures out what is going on here and fixes it, it would be VERY much appreciated.

carols10cents commented 12 years ago

I skipped the other one too.

carols10cents commented 12 years ago

UGH it looks like being able to find the delete link that has the text "I regret this" fails sometimes as well. I just skipped 2 more tests in acceptance/update_test.rb.

See Travis builds #129 and #135; those changes had nothing to do with being able to find the delete link.

carols10cents commented 12 years ago

Ok, there are more of these now, I've had it.

carols10cents commented 12 years ago

The new failures, and most of the old ones, appear to be tests that make external requests that would pass if you were using a previously-successful VCR cassette. That would explain why the failures are transient-- sometimes the external requests work and sometimes they don't.

I'm thinking about checking in the VCR cassettes I have in order to solve this problem. This introduces the possibility that we won't be notified through a test failure that an external site has changed, but I think the tradeoff of getting more stable test runs would be worth it.... Anyone have any thoughts before I do this?

One of the tests that we've skipped for a while does not use VCR; I'm trying a few travis runs that print out more of what the travis tests are seeing in my repo to try and figure that one out.

carols10cents commented 12 years ago

Adding the VCR cassettes seems to have helped some, but there are still 2 tests that fail a lot (but not every time) on travis, and never locally.

For serious, if anyone can fix this, name your reward. I will buy you coffee or beer. Or chocolate. Or pizza. Or an edible bouquet of fruit.

Spaceghost commented 12 years ago

Which tests are still broken on this? Any?

carols10cents commented 12 years ago

There are four pending tests that are marked with "skip" (they show up as "S" when you run the tests right now).

I actually saw a NEW one that is NOT skipped currently the other day on travis here: http://travis-ci.org/#!/hotsh/rstat.us/jobs/2509375

geopet commented 11 years ago

@carols10cents I'm trying to reproduce this bug and I'm not seeing the skips when running rake test. I'm likely doing something wrong here. Any suggestions? :cat:

jrgifford commented 11 years ago

Just saw this. https://gist.github.com/jrgifford/5070939

twopoint718 commented 11 years ago

Reporting as requested: https://gist.github.com/twopoint718/6373628

hindenbug commented 11 years ago

Encountered : https://gist.github.com/railsaholic/7184109 :confused: