R# Test Runner handles ignored tests differently that Gallio test runners (both Icarus and Echo)

[GoogleCodeExporter]

I will preface this by saying this is obscure, and one could argue breaks 
the atomic nature of a well written unit test. But, it is one of those 
edge cases that could expose one of those hard to pinpoint bugs. (I use 
this class in teaching students the danger of putting dependencies on 
other tests. The comments are specific to MbUnit2, I was working on 
updating them for MbUnit3/Gallio when I discovered this.)  
But it is a difference nonetheless.
What steps will reproduce the problem?
1. Reference MbUnit and Gallio 3.0.4.523
2. Create the following class:
using MbUnit.Framework;

namespace SampleTests.MBUnitDiscoveryTests
{
    /// <summary>
    /// This test fixture is used to demo some features of MbUnit
    /// </summary>
    public class LearnMBUnitTests
    {
        private int _FixtureSetUpCounter;
        private int _testSetUpCounter;

        [FixtureSetUp]
        public override void FixtureSetUp()
        {
            //Fires Once per test fixture before any tests have run
            _FixtureSetUpCounter++;
        }
        [SetUp]
        public void SetUp()
        {
            //Fires Once before EACH test, even ignored tests
            _testSetUpCounter++;
        }
        //In MbUnit, tests fire in the order they physically appear in the 
file
        [Test,Category("Learn MBUnit")]
        public void FirstTest()
        {
            Assert.AreEqual(1, _FixtureSetUpCounter);
            //Flipping the order in this class of FirstTest and SecondTest 
will cause these tests to fail
            Assert.AreEqual(1, _testSetUpCounter);
        }
        [Test,Ignore,Category("Learn MBUnit")]
        public void IgnoredTest()
        {
            Assert.AreEqual(1, _FixtureSetUpCounter);
            //Even though this test is ignored, the SetUp/TearDown methods 
run
            Assert.AreEqual(2, _testSetUpCounter);
        }
        [Test,Category("Learn MBUnit")]
        public void SecondTest()
        {
            Assert.AreEqual(1, _FixtureSetUpCounter);
            //This test is the second to run, the the third in the sequence
            //MbUnit3 fixed this issue
            Assert.AreEqual(2, _testSetUpCounter);
        }
        [TearDown]
        public void TearDown()
        {
            //Fires Once after EACH test, even ignored tests
        }

        [FixtureTearDown]
        public override void FixtureTearDown()
        {
            //Fires Once per test fixture after all tests have run
        }
    }
}
3. Execute the tests via R#. Test named SecondTest passes.
4. Execute via Gallio.Icarus or Gallio.Echo and the SecondTest fails.

What is the expected output? What do you see instead?
Expected output is the test named SecondTest should fail in all test 
harnesses or Pass in all test harnesses.

What version of the product are you using? On what operating system?
Gallio/MbUnit = 3.0.4.523
R# = 4.1.943.5
Vista 32bit

Please provide any additional information below.

Original issue reported on code.google.com by skime...@gmail.com on 13 Nov 2008 at 6:08

[GoogleCodeExporter]

Ok, I'm a little confused but I'll try it out.

A couple of comments:

1. Gallio will run tests by default in alphabetical order, not physical order.  
This
is done to provide a somewhat better user experience since GUI test runners 
often
show tests alphabetically.

(Note: A test framework can still force a non-deterministic order of execution 
if it
wants.)

2. By default, all tests have an Order index of 0.  You can change this in 
MbUnit v3
by adding the Order property to the [Test] or [TestFixture].  eg. 
[Test(Order=1)]. 
So actually Gallio runs tests (and fixtures) in order by increasing numeric 
index
proceeding alphabetically for tests (and fixtures) with the same order index.

3. It is possible to declare explicit test dependencies using the [DependsOn]
attribute in MbUnit v3.

Original comment by jeff.br...@gmail.com on 13 Nov 2008 at 6:37

[GoogleCodeExporter]

Close this issue.  Sorry.  I installed the latest daily build, and now I can't 
reproduce it. I will make a point of doing that FIRST next time.

Original comment by skime...@gmail.com on 13 Nov 2008 at 6:40

[GoogleCodeExporter]

Oh yes... and MbUnit v3 should NOT run SetUp or TearDown for Ignored tests.

However, [Ignore], [Explicit], and [Pending] tests, while skipped by default, 
will
nevertheless run if explicitly selected by the user.  This feature is intended 
to
make it easier for a user to troubleshoot an ignored test without having to 
make any
code changes first.

Original comment by jeff.br...@gmail.com on 13 Nov 2008 at 6:41

[GoogleCodeExporter]

Are you sure?

In any case, the R# runner uses the same code paths as the other runners to run
tests.  The only difference is that it's possible that the set of "explicitly
selected" tests may differ.  So for example, the ignored test might actually 
have run
in between FirstTest and SecondTest.  Consequently the setup and teardown would 
also
have run, causing the SecondTest assumption to fail.

May I suggest that a better demo of dependent tests would be to open up and 
write
some data to a file in one test then try to read it in another...  A bit more 
real-world.

Original comment by jeff.br...@gmail.com on 13 Nov 2008 at 6:44

[GoogleCodeExporter]

Thanks for the info. My class actually does that also through a base test 
fixture. I 
took that code out to make the example more simple and less code.  

I was definitely getting different results, but darned if I can reproduce that 
now.  
In Gallio.Icarus, the tests show up in FirstTest/IgnoredTest/SecondTest order.  
In 
R#, they show up as IgnoredTest/FirstTest/SecondTest.

It looks like R# is putting all of the ignored tests first, then proceeding.

Original comment by skime...@gmail.com on 13 Nov 2008 at 6:49

[GoogleCodeExporter]

Bottom line, the new gallio code is doing what it is supposed to.  I will 
modify my 
teaching tests to show that with the info you gave me. Thanks!

Original comment by skime...@gmail.com on 13 Nov 2008 at 6:53

[GoogleCodeExporter]

Unlikely that R# is running them in any different order.
However, it may be presenting them in a different order.

Order of presentation is not the same as order of execution.

Alright, I'm closing this issue.  Please let me know if you have any other 
questions
about this stuff.  I recommend the gallio-dev or mbunit-dev mailing lists.

Original comment by jeff.br...@gmail.com on 13 Nov 2008 at 7:19

• Changed state: Invalid