Pending tests not showing up correctly in Icarus

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Setup a few tests tagged as 'Pending', with Assert.Fail().
2. Select only the Pending tests and run.

** What is the expected output? What do you see instead?
Tests are expected to display in gray, indicating the 'Pending' status. The
'progress bar' is expected to include those pending tests as gray. Also,
the execution log should tag the pending tests correctly.

Now, they all show up as 'Failed'. The execution log shows the 'Pending
Reason: ...', but are marked red - 'Failed'.

** What version of the product are you using? On what operating system?
Gallio 3.0.6 build 763.

** Please provide any additional information below.
If I select and run the whole fixture, the tests show up correctly as Pending.

Original issue reported on code.google.com by mikael.r...@gmail.com on 7 Aug 2009 at 6:03

[GoogleCodeExporter]

I also noted now that the 'Pending' tests are not displayed with icons in the 
'Test
Results' list.

Original comment by mikael.r...@gmail.com on 7 Aug 2009 at 7:17

[GoogleCodeExporter]

This is by design.  If you explicitly run a Pending or Ignored test then the 
test
will run anyways so it might pass or fail depending on what it contains.

However, it would be good if Icarus automatically deselected Pending, Ignored 
and
Explicit tests when initially populating the test tree.

Original comment by jeff.br...@gmail.com on 9 Aug 2009 at 4:22

[GoogleCodeExporter]

Graham, let's see if we can improve how ignored/explicit/pending tests work in 
v3.2.
 I'll probably want to add some new kind of flag on tests to indicate they are
explicit.  Then Icarus can show the tests differently and leave them deselected 
by
default.

Original comment by jeff.br...@gmail.com on 11 Aug 2009 at 10:02

• Changed state: Accepted
• Added labels: Component-Icarus, Milestone-3.2, Priority-Low, Type-Enhancement

[GoogleCodeExporter]

Thanks for picking this up. As I see it, I never need a pending test reflecting 
the
actual test outcome (it would always fail). When I get back to finishing up a 
pending
test, the pending state has been resolved and so the pending tag should not be 
there
anymore.

Also, regardless who runs the test (it may not be the developer of the actual 
test
many times), it should be clear that it is pending on other tasks to complete 
without
looking at the test code. In our case, we have a testing department that will 
want to
run all tests as regression tests - if they see a failed test they will likely 
report
an error. Even if they will be unlikely to run anything but the whole test 
suite, it
should give the same result.

Original comment by mikael.r...@gmail.com on 12 Aug 2009 at 2:13

[GoogleCodeExporter]

Keep in mind that the problem is just with how Icarus selects tests. When 
running tests via Echo and 
other tools the pending tests will not run unless selected explicitly by the 
test filter.

Original comment by jeff.br...@gmail.com on 12 Aug 2009 at 2:47

[GoogleCodeExporter]

Icarus uses exactly the same default filter as Echo, if you select the root 
node then
Pending tests will not run. However if you explicitly select the tests (as in 
Step 2
above) then they will run, and will return whatever outcome you set up.

Original comment by grahamr...@gmail.com on 4 Sep 2009 at 9:44

[GoogleCodeExporter]

I've updated Icarus not to select Ignored/Pending tests by default, which makes 
it
more obvious, but will have no effect on which tests are actually run. 
Unfortunately
I don't think that's the answer the customer was looking for!

Original comment by grahamr...@gmail.com on 8 Oct 2009 at 7:35

[GoogleCodeExporter]

Original comment by Yann.Tre...@gmail.com on 17 Jul 2010 at 8:20

• Added labels: Milestone-3.3

[GoogleCodeExporter]

Original comment by Yann.Tre...@gmail.com on 14 Jun 2011 at 5:53

• Added labels: Milestone-IdeaPool