Test not running in order as stated in RootOrderedTestFixture.cs

[umarch06]

Hi, I have a scenario where I first need to order TestFixture and in each TestFixture I need Ordered Tests.

Here is the code:

using NUnitTestOrdering.FixtureOrdering;
using StarSoftware.Test.Tests.UI.Admin;
namespace StarSoftware.Test.NewFolder1
{
    [OrderedTestFixture]
    public sealed class MyOrderedTestFixture : TestOrderingSpecification
    {
        protected override void DefineTestOrdering()
        {
            this.TestFixture<CorporationTest>(config =>
             config.TestMethod(nameof(CorporationTest.UpdateCorporationDetails))
             .TestMethod(nameof(CorporationTest.ButtonClicks))
             );

            this.TestFixture<DivisionTest>(config =>
            config.TestMethod(nameof(DivisionTest.ClickExportButtonForDivision))
            );
        }

        protected override bool ContinueOnError => true; // true, if you want to continue even if a child test fails
    }
}

for showing how am I implementing DivisionTest, here it is:

using NUnit.Framework;
using NUnitTestOrdering.FixtureOrdering;
using NUnitTestOrdering.MethodOrdering;
using RelevantCodes.ExtentReports;
// my namespaces

namespace StarSoftware.Test.Tests.UI.Admin
{
    //[OrderedTestFixture]
    [TestFixture]
    public sealed class DivisionTest
    {
        #region private variables

        ExtentReports extent = PropertiesCollection.extent;
        ExtentTest test;

        #endregion

        #region Setup Test, this method run before test
        [SetUp]
        public void SetupTest()
        {
            Console.WriteLine("Division and Subdivision Tests");

            //navigate to admin division page
            SeleniumBrowserMethods.Navigate(TestUrls.AdminDivisionUrl);

            // start test and assign category
            test = extent.StartTest(TestContext.CurrentContext.Test.MethodName);
            test.AssignCategory(TestCategories.DivisionPageDivisionTests);
        }

        #endregion

        #region TearDown for this test

        [TearDown]
        public void TearDown()
        {

            ExtentReportsLog.LogError(test);
        }

        #endregion

        #region Tests

        #region Division Tests

        [Test, Category(TestCategories.DivisionPageDivisionTests)]
        public void DivisionGridAvailability()
        {
            bool isVisible = SeleniumObjectsMethods.IsElementVisible(PropertyType.Id, "DivisionGrid");
            Assert.AreEqual(isVisible, true);

            test.Log(LogStatus.Pass, "Division Grid is Visible");
        }

        #region Delete all divisions before test starts

        [Category(TestCategories.DivisionPageDivisionTests)]
        [Test]
        public void DeleteAllDivisions()
        {

        }

        #endregion

        #region division crud

        [Category(TestCategories.DivisionPageDivisionTests)]
        [Test, TestCaseSource(typeof(DivisionTestData), nameof(DivisionTestData.GetDivisionTestData))]
        public void CreateDivision(DivisionViewModel model)
        {

        }

        [Category(TestCategories.DivisionPageDivisionTests)]
        [Test, TestCaseSource(typeof(DivisionTestData), nameof(DivisionTestData.GetDivisionTestData))]
        public void UpdateDivision(DivisionViewModel model)
        {

        }

        [Category(TestCategories.DivisionPageDivisionTests)]
        [Test, TestCaseSource(typeof(DivisionTestData), nameof(DivisionTestData.GetDivisionTestData))]
        public void DeleteDivision(DivisionViewModel model)
        {

        }

        #endregion

        [Category(TestCategories.DivisionPageDivisionTests)]
        [Test]
        public void ClickExportButtonForDivision()
        {

        }

        #endregion

        #region Subdivison Tests

        [Category(TestCategories.DivisionPageSubDivisionTests)]
        [Test]
        public void SubDivisionGridAvailability()
        {

        }

        [Category(TestCategories.DivisionPageSubDivisionTests)]
        [Test]
        public void ClickExportButtonForSubDivision()
        {

        }

        #region sub division crud

        [Category(TestCategories.DivisionPageSubDivisionTests)]
        [Test]
        [TestCaseSource(typeof(SubDivisionTestData), nameof(SubDivisionTestData.GetSubDivisionTestData))]
        public void CreateSubDivision(SubDivisionViewModel model)
        {

        }

        [Category(TestCategories.DivisionPageSubDivisionTests)]
        [Test]
        [TestCaseSource(typeof(SubDivisionTestData), nameof(SubDivisionTestData.GetSubDivisionTestData))]
        public void RetrieveSubDivision(SubDivisionViewModel model)
        {

        }

        [Category(TestCategories.DivisionPageSubDivisionTests)]
        [Test]
        [TestCaseSource(typeof(SubDivisionTestData), nameof(SubDivisionTestData.GetSubDivisionTestData))]
        public void UpdateSubDivision(SubDivisionViewModel model)
        {
        }

        [Category(TestCategories.DivisionPageSubDivisionTests)]
        [Test]
        [TestCaseSource(typeof(SubDivisionTestData), nameof(SubDivisionTestData.GetSubDivisionTestData))]
        public void DeleteSubDivision(SubDivisionViewModel model)
        {
        }

        #endregion

        #endregion

        #endregion

    }
}

CorportationTest and DivisionTest are two tests fixtures. Although I have specified only ClickExportForDivision in code but it is executing all the tests in CorporationTest TestFixture class. And Order is not working correct. Am I missing something? This is what I get when I build the project:

And when I run all tests, all the tests run in the order in which it was previously running. DefineTestOrdering did not affect anything!

[Sebazzz]

If I may summarize, the problem is that too many tests are run and no ordering appears to be applied.

Please supply the following information:

• NUnit version
• NUnitTestOrdering version

Please try to following too:

• Check whether the EnableTestFixtureOrdering attribute has been applied to the assembly as per instructions.
• Try using a different test runner, like the NUnit GUI (in alpha stage), or the NUnit console.

[umarch06]

I am on NUnit 3.6.1 and NUnitTestOrdering 1.0.0

• I tested the dll in NUnit GUI alpha for NUnit 3. but same output.
• And on NUnit console, Test does not run and following xml is generated without any test execution at following path C:\Program Files (x86)\NUnit.org\nunit-console\TestResult.xml

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-run id="2" testcasecount="52" result="Passed" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.6.1.0" clr-version="4.0.30319.42000" start-time="2017-05-24 09:46:28Z" end-time="2017-05-24 09:46:29Z" duration="1.051725">
  <command-line><![CDATA[nunit3-console.exe  "C:\Users\Umar Chaudhry\Desktop\Test\MetisLatest\QA\StarSoftware.Test\bin\Release\StarSoftware.Test.dll" --where "test==Ordered.RootOrderedTestFixture.OrderedTestFixture1"]]></command-line>
  <filter>
    <test>Ordered.RootOrderedTestFixture.OrderedTestFixture1</test>
  </filter>
  <test-suite type="Assembly" id="0-1064" name="StarSoftware.Test.dll" fullname="C:\Users\Umar Chaudhry\Desktop\Test\MetisLatest\QA\StarSoftware.Test\bin\Release\StarSoftware.Test.dll" runstate="Runnable" testcasecount="52" result="Passed" start-time="2017-05-24 09:46:29Z" end-time="2017-05-24 09:46:29Z" duration="0.012920" total="0" passed="0" failed="0" warnings="0" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.6.1.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 10.0.14393.0" platform="Win32NT" cwd="C:\Program Files (x86)\NUnit.org\nunit-console" machine-name="BROOM" user="Umar Chaudhry" user-domain="BROOM" culture="en-GB" uiculture="en-GB" os-architecture="x64" />
    <settings>
      <setting name="DisposeRunners" value="True" />
      <setting name="WorkDirectory" value="C:\Program Files (x86)\NUnit.org\nunit-console" />
      <setting name="ImageRuntimeVersion" value="4.0.30319" />
      <setting name="ImageTargetFrameworkName" value=".NETFramework,Version=v4.5" />
      <setting name="ImageRequiresX86" value="False" />
      <setting name="ImageRequiresDefaultAppDomainAssemblyResolver" value="False" />
      <setting name="NumberOfTestWorkers" value="8" />
    </settings>
    <properties>
      <property name="_PID" value="21140" />
      <property name="_APPDOMAIN" value="domain-" />
    </properties>
  </test-suite>
</test-run>

[Sebazzz]

Could you create a dump of your test assembly using the --explore option. I'd like both testcases and NUnit 3 format. You can e-mail it to me privately if you don't like it on Github.

edit: Thanks, got the mail.

[umarch06]

I sent you in email! Thanks

[umarch06]

@Sebazzz Did you get it working?

[Sebazzz]

Unfortunately not yet, sorry.

[Sebazzz]

Sorry it took so long. It appears you have forgotten to apply the EnableTestOrderingAttribute as per documentation.

I loaded up your assembly in Powershell and checked the attributes:

$Assembly = [System.Reflection.Assembly]::LoadFrom($(Resolve-Path [redacted].Test.dll).Path)
$Assembly.CustomAttributes | ? AttributeType -notlike System*

Output of the command above is nothing. Expected output would be something like this:

AttributeType                                                 Constructor  ConstructorArguments NamedArguments
-------------                                                 -----------  -------------------- --------------
NUnitTestOrdering.FixtureOrdering.EnableTestOrderingAttribute Void .ctor() {}                   {}

This shows the EnableTestOrderingAttribute is applied. If you use the latest master the library will also attach a custom property to the assembly which is outputted during an --explore in XML format or in the test results XML.

Try applying the attribute and let me know how it goes. Please note: I recommend using NUnit command line runner or NUnit gui.

[umarch06]

Hi, thanks for your update. I had been waiting for so long. Everything is working now problem is different for me.

When I need OneTimeSetUp and OnTimeTearDown for multiple fixtures like that I am doing here.

Fixture1

using NUnit.Framework;

namespace NUnitDemo
{
    [TestFixture]
    public class Fixture1
    {
        #region setup and teardown, it is same for all tests in Fixture1

        [SetUp]
        public void SetupTest()
        {
            Logger.Log("SetupTest of all tests in Fixture 1.");
        }

        [TearDown]
        public void TearDown()
        {
            Logger.Log("TearDown of all tests in Fixture 1.");
        }

        #endregion

        [Test]
        [Category("Fixture1")]
        public void Fixture1Test1()
        {
            Logger.Log("Fixture1Test1");
        }

        [Test]
        [Category("Fixture1")]
        public void Fixture1Test2()
        {
            Logger.Log("Fixture1Test2");
        }

        [Category("Fixture1")]
        [Test]
        public void Fixture1Test3()
        {
            Logger.Log("Fixture1Test3");
        }

        [Category("Fixture1")]
        [Test]
        [TestCaseSource(typeof(TestCases), nameof(TestCases.Tests))]
        public void Fixture1Test4(string input)
        {
            Logger.Log(input);
        }

    }
}

Fixture 2

using NUnit.Framework;

namespace NUnitDemo
{
    [TestFixture]
    public class Fixture2 : GlobalSetupTearDown
    {

        #region setup and teardown, it is same for all tests in Fixture2

        [SetUp]
        public void SetupTest()
        {
            Logger.Log("SetupTest of all tests in Fixture 2.");
        }

        [TearDown]
        public void TearDown()
        {
            Logger.Log("TearDown of all tests in Fixture 2.");
        }

        #endregion

        [Category("Fixture2")]
        [Test]
        public void Fixture2Test1()
        {
            Logger.Log("Fixture2Test1");
        }

        [Category("Fixture2")]
        [Test]
        public void Fixture2Test2()
        {
            Logger.Log("Fixture2Test2");
        }

        [Category("Fixture2")]
        [Test]
        public void Fixture2Test3()
        {
            Logger.Log("Fixture2Test3");
        }
    }

}

GlobalSetupTearDown

using NUnit.Framework;
using NUnitTestOrdering.FixtureOrdering;

namespace NUnitDemo
{
    [SetUpFixture]
    public class GlobalSetupTearDown : TestOrderingSpecification
    {

        [OneTimeSetUp]
        public void GlobalSetup()
        {
            Logger.Log("Started All Tests");
        }

        [OneTimeTearDown]
        public void GlobalTearDown()
        {
            Logger.Log("Finished All Tests");
        }

        protected override void DefineTestOrdering()
        {

        }

        protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails

    }
}

MyOrderedTestFixture

using NUnitTestOrdering.FixtureOrdering;

namespace NUnitDemo
{
    [OrderedTestFixture]
    public sealed class MyOrderedTestFixture :  GlobalSetupTearDown
    {

        protected override void DefineTestOrdering()
        {
            TestFixture<Fixture1>(config => config.TestMethod(nameof(Fixture1.Fixture1Test3))
            .TestMethod(nameof(Fixture1.Fixture1Test1))
            .TestMethod(nameof(Fixture1.Fixture1Test4))
            );

            TestFixture<Fixture2>(config => config.TestMethod(nameof(Fixture2.Fixture2Test3)));
        }

        protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails
    }
}

TestCases

using System.Collections.Generic;

namespace NUnitDemo
{
    public static class TestCases
    {
        public static IEnumerable<string> Tests()
        {
            yield return "Source 1";
            yield return "Source 2";
            yield return "Source 3";
        }
    }
}

I am inheriting GlobalSetupTearDown from TestOrderingSpecification, the oneTimeTearDown and OneTimeSetup should must be call at beginning no matter how many test fixtures I use but it is not doing so. OneTimeSetup and OneTimeTearDown is called for each fixture. Can you please help?

Here is output I get:

Started All Tests
SetupTest of all tests in Fixture 1.
Fixture1Test3
TearDown of all tests in Fixture 1.
SetupTest of all tests in Fixture 1.
Fixture1Test1
TearDown of all tests in Fixture 1.
SetupTest of all tests in Fixture 1.
Source 1
TearDown of all tests in Fixture 1.
SetupTest of all tests in Fixture 1.
Source 2
TearDown of all tests in Fixture 1.
SetupTest of all tests in Fixture 1.
Source 3
TearDown of all tests in Fixture 1.
Started All Tests
SetupTest of all tests in Fixture 2.
Fixture2Test3
TearDown of all tests in Fixture 2.
Finished All Tests
Finished All Tests

It seems like it treats oneTimeSetup and Tear down as nested Global methods instead of considering them same for each fixture.

[Sebazzz]

It is because you inherit from the GlobalSetupTearDown. The fixture level methods run for the ordering specification - because it inherits from GlobalSetupTearDown and run for each fixture because they inherit GlobalSetupTearDown.

If you want the methods to run once for the entire hierarchy, just let the MyOrderedTestFixture inherit from GlobalSetupTearDown and do not inherit your actual test fixtures from that class.

[umarch06]

Thanks so much, sorry I was inheriting Fixture2 from GlobalSetupTearDown, this was un intentional. I am curious about multiple fixture support for this scenario? I don't know what to do for that case?

This is what I need:

MyOtherOrderedTestFixture

using NUnitTestOrdering.FixtureOrdering;

namespace NUnitDemo
{
    [OrderedTestFixture]
    public class MyOtherOrderedTestFixture : MyOrderedTestFixture //this is beacuse I needed to override 
    //Define TestOrdering
    {
        protected override void DefineTestOrdering()
        {
            Logger.Log("I am in OrderedTestFixture1");

            TestFixture<Fixture2>(config => config.TestMethod(nameof(Fixture2.Fixture2Test3)));

            TestFixture<Fixture2>(config => config.TestMethod(nameof(Fixture2.Fixture2Test3)));
            TestFixture<Fixture1>(config => config.TestMethod(nameof(Fixture1.Fixture1Test2)));
            Logger.Log("I am out from OrderedTestFixture1");
        }

        protected override bool ContinueOnError => false;
    }
}

and here is what I need. basically I want to use multiple OrderedTestFixtures with same GlobalSetupTearDown. I can't go with it unless I derive the MyOtherOrderedTestFixture from something which inherits TestOrderingSpecification.

Here is updated MyOrderedTestFixture:

[OrderedTestFixture]
    public class MyOrderedTestFixture :  GlobalSetupTearDown
    {

        protected override void DefineTestOrdering()
        {
            OrderedTestSpecification<MyOtherOrderedTestFixture>();

            TestFixture<Fixture1>(config => config.TestMethod(nameof(Fixture1.Fixture1Test3))
            .TestMethod(nameof(Fixture1.Fixture1Test1))
            .TestMethod(nameof(Fixture1.Fixture1Test4))
            );

            TestFixture<Fixture2>(config => config.TestMethod(nameof(Fixture2.Fixture2Test3)));

            TestFixture<Fixture2>(config => config.TestMethod(nameof(Fixture2.Fixture2Test3)));
            TestFixture<Fixture1>(config => config.TestMethod(nameof(Fixture1.Fixture1Test2)));

        }

        protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails
    }

[Sebazzz]

If you refer another ordered test fixture the OneTimeSetup/OneTimeTearDown methods will also execute on that one.

From a quick glance, I'd say the ordering would be:

MyOrderedTestFixture.OneTimeSetup
MyOtherOrderedTestFixture.OneTimeSetup
(tests in MyOtherOrderedTestFixture)
MyOtherOrderedTestFixture .OneTimeTearDown
(MyOrderedTestFixture tests)
MyOrderedTestFixture.OneTimeTearDown

Please also look at the tests in this repository. They should give a good idea on what is possible with the library. And I recommend the NUnit GUI (which is in alpha stage) to visualize the testing hierarchy.

[umarch06]

I fixed this by inheriting MyOtherOrderedTestFixture from TestOrderingSpecification. Everything is working fine! Thank you legend! Closing the issue 👍

[umarch06]

Hi sorry for reopening. In my last comment, I am facing a problem. Since I did not inherit MyOtherOrderedTestFixture from GlobalSetupTearDown where as doing from TestOrderSpecificaiton, I would not be able to test each fixture separately. I also used [OrderedTestGlobalSetUpFixture] from this example for global setup and tear down but it is also not working. (Note: tried this example for both single and multiple OrderedTestFixture).

[Sebazzz]

If OrderedTestGlobalSetUpFixture does not work for you, please create a minimum example in which the bug repro's and upload it to GitHub Gist. Please be specific in what kind of behavior you expect and what currently happens. Thanks

[umarch06]

Fixture1

using NUnit.Framework;

namespace NUnitDemo
{
    [TestFixture]
    public class Fixture1
    {
        #region setup and teardown, it is same for all tests in Fixture1

        [SetUp]
        public void SetupTest()
        {
            //Logger.Log("SetupTest of all tests in Fixture 1.");
        }

        [TearDown]
        public void TearDown()
        {
            //Logger.Log("TearDown of all tests in Fixture 1.");
        }

        #endregion

        [Test]
        [Category("Fixture1")]
        public void Fixture1Test1()
        {
            Logger.Log("Fixture1Test1");
        }

        [Test]
        [Category("Fixture1")]
        public void Fixture1Test2()
        {
            Logger.Log("Fixture1Test2");
        }

        [Category("Fixture1")]
        [Test]
        public void Fixture1Test3()
        {
            Logger.Log("Fixture1Test3");
        }

        [Category("Fixture1")]
        [Test]
        [TestCaseSource(typeof(TestCases), nameof(TestCases.Tests))]
        public void Fixture1Test4(string input)
        {
            Logger.Log(input);
        }

    }
}

Fixture2

using NUnit.Framework;

namespace NUnitDemo
{
    [TestFixture]
    public class Fixture2
    {

        #region setup and teardown, it is same for all tests in Fixture2

        [SetUp]
        public void SetupTest()
        {
            //Logger.Log("SetupTest of all tests in Fixture 2.");
        }

        [TearDown]
        public void TearDown()
        {
            //Logger.Log("TearDown of all tests in Fixture 2.");
        }

        #endregion

        [Category("Fixture2")]
        [Test]
        public void Fixture2Test1()
        {
            Logger.Log("Fixture2Test1");
        }

        [Category("Fixture2")]
        [Test]
        public void Fixture2Test2()
        {
            Logger.Log("Fixture2Test2");
        }

        [Category("Fixture2")]
        [Test]
        public void Fixture2Test3()
        {
            Logger.Log("Fixture2Test3");
        }
    }

}

OrderedFixtureTests1

using NUnitTestOrdering.FixtureOrdering;

namespace NUnitDemo
{
    [OrderedTestFixture]
    public class OTF1 : TestOrderingSpecification
    {
        protected override void DefineTestOrdering()
        {
            TestFixture<Fixture2>(config => config.TestMethod(nameof(Fixture2.Fixture2Test3)));
            TestFixture<Fixture1>(config => config.TestMethod(nameof(Fixture1.Fixture1Test2))
            .TestMethod(nameof(Fixture1.Fixture1Test1))
            );
        }

        protected override bool ContinueOnError => false;
    }
}

OrderedTestFixture2

using NUnitTestOrdering.FixtureOrdering;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace NUnitDemo
{
    [OrderedTestFixture]
    public class OTF2 : TestOrderingSpecification
    {
        protected override void DefineTestOrdering()
        {
            TestFixture<Fixture2>(config => config.TestMethod(nameof(Fixture2.Fixture2Test1)));
            TestFixture<Fixture1>(config => config.TestMethod(nameof(Fixture1.Fixture1Test2)));
        }

        protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails

    }
}

MyOrderedTestFixture

using NUnitTestOrdering.FixtureOrdering;

namespace NUnitDemo
{
    [OrderedTestFixture]
    public class MyOrderedTestFixture : TestOrderingSpecification
    {
        protected override void DefineTestOrdering()
        {
            OrderedTestSpecification<OTF1>();
            OrderedTestSpecification<OTF2>();
        }

        protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails
    }
}

GlobalSetupFixture

using NUnit.Framework;
using NUnitTestOrdering.FixtureOrdering.Internal;

namespace NUnitDemo
{
    [SetUpFixture]
    [OrderedTestGlobalSetUpFixture]
    public sealed class GlobalSetUpFixture
    {
        [OneTimeSetUp]
        public void SetUp()
        {
            Logger.Log("Started All Tests");
        }

        [OneTimeTearDown]
        public void TearDown()
        {
            Logger.Log("Finished All Tests");
        }

    }
}

I run MyOrderedTestFixture

Output:

Fixture2Test3 Fixture1Test2 Fixture1Test1 Fixture2Test1 Fixture1Test2

Expected output:

Started All Tests Fixture2Test3 Fixture1Test2 Fixture1Test1 Fixture2Test1 Fixture1Test2 Finished All Tests

[Sebazzz]

Thank you for your clear test case. Please try putting the GlobalSetUpFixture in the global namespace (no namespace declaration) like the example.

[umarch06]

It worked. 🥇