Tests not reported as Skipped or Inconclusive if there is an error during an AssemblyFixture SetUp or FixtureSetUp method.

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Create a MbUnit 3 testing project with an AssemblyFixture 
2. Cause a failure in either the [SetUp] method or [FixtureSetUp] method of
the AssemblyFixture
3. Have tests in other TestFixtures

What is the expected output? What do you see instead?
The test runners (Gallio Icarus, Gallio Echo, and when run through team
city (which I assume is build on Echo) will report results of 0 run, 0
passed, 0 failed, 0 inconclusive, 0 skipped; I would expect that it would
report the tests in the test fixtures as Skipped (or perhaps inconclusive)
instead of pretending that they do not exist.

What version of the product are you using? On what operating system?

3.0.6 Update 1; On both Windows XP and Windows Server 2003.

Please provide any additional information below.

If some tests have already run before the failure during the
AssemblyFixture's SetUp method those will be reported properly - but other
tests for the following fixture will not be reported at all.

Original issue reported on code.google.com by mab...@gmail.com on 14 Apr 2009 at 10:55

[GoogleCodeExporter]

That's not good.  Let's see what we can do for v3.0.6 Update 2.

Original comment by jeff.br...@gmail.com on 27 Apr 2009 at 6:08

• Changed state: Accepted
• Added labels: Component-MbUnit, Milestone-3.0.6, Priority-High, Type-Defect

[GoogleCodeExporter]

Ok, so in general the problem is that if fixture setup or teardown fails then 
the
children in the test tree just don't ever get a chance to run.  This is a 
problem not
just for assembly fixtures and test fixture but other things too, like suites.

I guess the solution will either be to generate fake failure results for tests 
that
did not run or to promote the fixture to "test case" status so it shows as a 
failure
in the stats.

The problem is that the statistics will still be wrong when any data-driven 
tests
were not run.  For example, if all N runs of a data-driven test (eg. row-test) 
failed
you would see N failures logged.  However if an assembly fixture fails 
initialization
then in general we cannot enumerate the whole list of runs there would have 
been so
we don't know how many failures to report...

Consequently, I'm thinking I might need to introduce a new TestStatus like 
"blocked"
or "fatal".  The idea here is that when we cannot initialize something like an
assembly fixture, we just give up on that whole part of the test tree and 
report the
meta-failure loudly.

In any case, fixing this is too risky for v3.0.6 so I'll defer it in v3.0.7 
instead.

Original comment by jeff.br...@gmail.com on 28 Apr 2009 at 3:31

• Added labels: Milestone-3.0.7
• Removed labels: Milestone-3.0.6

[GoogleCodeExporter]

Note that marking a fixture [Ignored] also has similar effects: we don't get 
separate
ignored results in the stats for each test in the fixture...

Original comment by jeff.br...@gmail.com on 28 Apr 2009 at 3:33

[GoogleCodeExporter]

This sounds like a good way to fix it; I wasn't particularly concerned about 
the numbers being correct; it is more 
that it doesn't report any failure at all - which is very problematic for 
continuous integration as it doesn't detect 
any problem and we don't get alerted, and it treats it as a success; when it 
could be (and likely is) a serious issue 
if the AssemblyFixture cannot be run.

Original comment by mab...@gmail.com on 30 Apr 2009 at 9:06

[GoogleCodeExporter]

Original comment by jeff.br...@gmail.com on 29 Jun 2009 at 8:33

• Added labels: Milestone-3.1

[GoogleCodeExporter]

Let's see what we can do.

Original comment by jeff.br...@gmail.com on 31 Jul 2009 at 10:44

[GoogleCodeExporter]

The fix now is to promote a fixture to the status of "test case" if it fails and
appears to have prevented its child tests from failing.

Bottom line, you will see a failure in the report statistics now.

Original comment by jeff.br...@gmail.com on 14 Aug 2009 at 11:32

• Changed state: Fixed