SetupFixture method having an exception leads to "Duplicates not allowed" error or inconsistency.

[IgorKaplya]

I'll provide a minimal reproducible example below.

For the context: when the [SetupFixture] method raises an exception

• Tests still attempt to be executed.
  • GUI runner breaks with the "Duplicates not allowed" exception.
  • UI remains disabled till the test app restart.
  • No next fixtures processed.
  • Console runner ends up with 2 tests passed and the same 2 tests errored
  • This is inconsistent there are only 2 tests. And those are not Schrödinger's cat to be passed and errored at the same time.

I've debugged a bit and assume that the reason is that TDUnitXTestRunner.InvalidateTestsInFixture that is called on exception of the TDUnitXTestRunner.ExecuteSetupFixtureMethod should, for example, set all fixtures tests enabled property to false. In this case test execution will be skipped (Option A). But TearDownFixture() still will be called. Otherwise it makes sense to interrupt next operations after the failed SetupFixture (Option B).

Here is the minimal reproducible example:

DPR:

(* Uncomment to see GUI runner behaviour
program Project1;

uses
  Vcl.Forms,
  SysUtils,
  DUnitX.Loggers.GUI.VCL,
  DUnitX.TestFramework,
  Unit1 in 'Unit1.pas';

begin
  Application.Initialize;
  Application.CreateForm(TGUIVCLTestRunner, GUIVCLTestRunner);
  Application.Run;
end.
*)

program Project1;

{$IFNDEF TESTINSIGHT}
{$APPTYPE CONSOLE}
{$ENDIF}
{$STRONGLINKTYPES ON}
uses
  System.SysUtils,
  {$IFDEF TESTINSIGHT}
  TestInsight.DUnitX,
  {$ELSE}
  DUnitX.Loggers.Console,
  DUnitX.Loggers.Xml.NUnit,
  {$ENDIF }
  DUnitX.TestFramework,
  Unit1 in 'Unit1.pas';

{$IFNDEF TESTINSIGHT}
var
  runner: ITestRunner;
  results: IRunResults;
  logger: ITestLogger;
  nunitLogger : ITestLogger;
{$ENDIF}
begin
{$IFDEF TESTINSIGHT}
  TestInsight.DUnitX.RunRegisteredTests;
{$ELSE}
  try
    //Check command line options, will exit if invalid
    TDUnitX.CheckCommandLine;
    //Create the test runner
    runner := TDUnitX.CreateRunner;
    //Tell the runner to use RTTI to find Fixtures
    runner.UseRTTI := True;
    //When true, Assertions must be made during tests;
    runner.FailsOnNoAsserts := False;

    //tell the runner how we will log things
    //Log to the console window if desired
    if TDUnitX.Options.ConsoleMode <> TDunitXConsoleMode.Off then
    begin
      logger := TDUnitXConsoleLogger.Create(TDUnitX.Options.ConsoleMode = TDunitXConsoleMode.Quiet);
      runner.AddLogger(logger);
    end;
    //Generate an NUnit compatible XML File
    nunitLogger := TDUnitXXMLNUnitFileLogger.Create(TDUnitX.Options.XMLOutputFile);
    runner.AddLogger(nunitLogger);

    //Run tests
    results := runner.Execute;
    if not results.AllPassed then
      System.ExitCode := EXIT_ERRORS;

    {$IFNDEF CI}
    //We don't want this happening when running under CI.
    if TDUnitX.Options.ExitBehavior = TDUnitXExitBehavior.Pause then
    begin
      System.Write('Done.. press <Enter> key to quit.');
      System.Readln;
    end;
    {$ENDIF}
  except
    on E: Exception do
      System.Writeln(E.ClassName, ': ', E.Message);
  end;
{$ENDIF}
end.

Test unit:

unit Unit1;

interface

uses
  DUnitX.TestFramework;

type
  [TestFixture]
  TMyTestObject = class
  public
    [SetupFixture]
    procedure SetupFixture;
    [Setup]
    procedure Setup;
    [Test]
    procedure Test1();
    [Test]
    procedure Test2();
  end;

implementation

uses
  System.SysUtils, Vcl.Dialogs;

procedure TMyTestObject.Setup;
begin
  Log('Setup');
  // raise Exception.Create('Error in Setup');
end;

procedure TMyTestObject.SetupFixture;
begin
  raise Exception.Create('Error SetupFixture');
end;

procedure TMyTestObject.Test1();
begin
  Assert.Pass('Test1');
end;

procedure TMyTestObject.Test2();
begin
  Assert.Pass('Test2');
end;

initialization
  TDUnitX.RegisterTestFixture(TMyTestObject);

end.