Declare Tests at Runtime

[zedalaye]

I want to build an extensive test suite to cover all possible "variants" for an algorithm. I ended declaring my "Test Cases" in an External JSON file and my "RunTests" methods just read that file, instanciates an instance of the "algorithm" and passes all declared test cases. The problem is that withing DUnitX GUI VCL Runner, all test cases are "hidden" behind a single line for the whole "Test Suite".

Is it possible to declare such TestCases at runtime so ther are reported individually ?

[zedalaye]

Okay, I found a way to do that, for those interrested:

unit DUnitX.Example.DynamicTestCases;

interface

uses
  SysUtils, Classes, Rtti,
  superobject,
  DUnitX.TestFramework,
  DUnitX.Extensibility,
  DUnitX.Types;

type
  TJSONFixturesProviderPlugin = class(TInterfacedObject, IPlugin)
    procedure GetPluginFeatures(const context: IPluginLoadContext);
  end;

  TJSONFixturesProvider = class(TInterfacedObject, IFixtureProvider)
  private class var
    FRttiContext: TRttiContext;
  protected
    procedure Execute(const context: IFixtureProviderContext);
  public
    class constructor InitContext;
    class destructor FreeContext;
  end;

  TDynamicFixture = class
  public
    procedure RunTest(const ArgsJSON: string);
  end;

implementation

{ TDynamicFixture }

procedure TDynamicFixture.RunTest(const ArgsJSON: string);
var
  params: ISuperObject;
begin
  params := TSuperObject.ParseString(PChar(ArgsJSON), True);
  Assert.AreEquals(params.I[test], 1);
end;

{ TJSONFixturesProviderPlugin }

procedure TJSONFixturesProviderPlugin.GetPluginFeatures(
  const context: IPluginLoadContext);
begin
  context.RegisterFixtureProvider(TJSONFixturesProvider.Create);
end;

{ TJSONFixturesProvider }

procedure TJSONFixturesProvider.Execute(const context: IFixtureProviderContext);
var
  RttiType: TRttiType;
  RttiMethod: TRttiMethod;

  TestFileName: string;
  TestFileStream: TStream;
  TestBytes: TBytes;
  TestData: string;
  TestSuite, TestCase: ISuperObject;

  F: ITestFixture;
  TestParams: TValueArray;
begin
  TestFileName := ExpandFileName(ExtractFilePath(ParamStr(0)) + '..\Tests\test_cases.json');
  if not FileExists(TestFileName) then
    Exit;

  { Workaround because SuperObject do not know how to load UTF8 encoded files }
  TestFileStream := TFileStream.Create(TestFileName, fmOpenRead);
  try
    SetLength(TestBytes, TestFileStream.Size);
    TestFileStream.ReadBuffer(TestBytes[0], TestFileStream.Size);
    TestData := TEncoding.UTF8.GetString(TestBytes)
  finally
    TestFileStream.Free;
  end;

  TestSuite := TSuperObject.ParseString(PChar(TestData), True);
  if ObjectIsType(TestSuite, stArray) and (TestSuite.AsArray.Length > 0) then
  begin
    RttiType := FRttiContext.GetType(TDynamicFixture);
    RttiMethod := RttiType.GetMethod('RunTest');

    F := context.CreateFixture(TObject, 'TDynamicFixture', '');
    F := F.AddChildFixture(TDynamicFixture, 'TDynamicFixture', '');
    for TestCase in TestSuite do
    begin
      SetLength(TestParams, 1);
      TestParams[0] := TValue.From<string>(TestCase.AsJson(false, false));
      F.AddTestCase('RunTest', TestCase.S['desc'], 'RunTest', '', RttiMethod, True, TestParams);
    end;
  end;
end;

class constructor TJSONFixturesProvider.InitContext;
begin
  FRttiContext := TRttiContext.Create;
end;

class destructor TJSONFixturesProvider.FreeContext;
begin
  FRttiContext.Free;
end;

initialization
  TDUnitX.RegisterPlugin(TJSONFixturesProviderPlugin.Create);

end.

File containing Test Cases :

[
  { "desc": "Test case 1", "test": 1 },
  { "desc": "Test case 2", "test": 2 }
]

Hope this helps :) I just don't know how to "correctly" (read : like TDUnitXFixtureProvider.Execute do) register "tree" of fixtures (parent/child fixtures) so it looks like they have been discovered automagically. May you can help me to improve that ?

[UweRupprecht]

Just finished the work on an a general way, to dynamically create Testcases with the use of external Data (Files, Database..or what ever you want). See the pull request :)

[rpottsoh]

This is intriguing to me as is what @UweRupprecht is working on. I maintain the Delphi track at exercism.io. All of the exercises that are available to all of the different language tracks are written in JSON. A few of the tracks have written test generators to translate the JSON into test suites for their particular language. For now I have been translating the JSON manually into DUnitX test suites. If what you two have been working on might remove (reduce) the need to translate the JSON.... I am interested. If you are curious, the JSON is stored here https://github.com/exercism/problem-specifications/tree/master/exercises

[UweRupprecht]

Well, may be in some way.

You have to write a class, that processes the data accordingly for using it to create test cases. Then the class must be registered to a Managerclass (or Factoryclass).

When this is done, you can just use [TestCaseProvider('YourProviderClass')} instead of [TestCase('Case 01','x,y,z')] [TestCase('Case 02','x,y,z')] [TestCase('Case 03','x,y,z')] [TestCase('Case 04','x,y,z')] [TestCase('Case 05','x,y,z')].......

[vincentparrett]

@zedalaye We will most likely go with @UweRupprecht pull request (just some minor tweaking needed) so closing this case.