Test run not failing when setup fails.

[GoogleCodeExporter]

We are observing misleading statistics when a setup fails.  A fixture setup
method failure causes the fixture to be marked as a failure.  However a
fixture is not considered a test case so its failure is not represented in
the overall statistics.  This is a problem in general for all
non-successful outcomes arising from inner nodes of the test tree.

Unfortunately we cannot simply tally up all failed tests within the fixture
because a fixture setup failure might actually prevent us from enumerating
those test cases correctly (particularly when data binding is involved).

Alternatives:

1. Compute statistics over test cases and all leaves that are not
descendants of test cases.

2. Introduce new statistics to indicate when interior nodes cause test
cases not to run.  eg. a "blocked" status.  However it is unclear how we
would distinguish blockage due to failure vs. due to being ignored.

Original issue reported on code.google.com by jeff.br...@gmail.com on 17 Jun 2009 at 6:33

[GoogleCodeExporter]

Original comment by jeff.br...@gmail.com on 29 Jun 2009 at 8:33

• Added labels: Milestone-3.1

[GoogleCodeExporter]

Now we promote failing fixtures and other interior nodes of the test tree to 
"test
case" status so they show up in the report if they fail and prevent other tests 
from
running.

Original comment by jeff.br...@gmail.com on 16 Aug 2009 at 4:55

• Changed state: Fixed