search

Yelp / Testify

A more pythonic testing framework.
Other
308 stars 67 forks source link

more buckets than tests is not an error #165

Closed mrtyler closed 8 years ago

mrtyler commented 11 years ago

We had a situation today where non_sandbox builders 2 and 5 on buildbot6 never ran any tests, therefore throwing an error.

Upon investigation, we discovered this is "expected" -- by stealing the Testify command line from the two empty builders, we got Testify to tell us there were no tests in those buckets. The expected number of tests showed up in other buckets.

Testify currently flags this situation as an error:

========================================================================
FAILURES
None!

No tests were discovered (tests must subclass TestCase and test methods must begin with 'test').
ERROR.  0 tests / 0 cases: 0 passed, 0 failed.  (Total test time 0.00s)
program finished with exit code 1

After discussion, we don't think this is an error condition. It would be nice if the bucketing algorithm were better (see Issue #166) but regardless the existence of an empty bucket is not an error.

asottile commented 10 years ago

From #172

The current (newish) bucketing algorithm should avoid empty buckets if there are at least N test case classes where N is the number of buckets. I don't think this resolves this issue though.

bukzor commented 10 years ago

Correct, it just makes the problem more rare.

asottile commented 8 years ago

Via #328