C# test generator

[dvvrd]

For now, V# supports only XML-based unit test representation. Besides unreadability issues, we need the special tool for running these tests. That is, given a collection of UnitTest objects, currently we print them using XmlSerializer.

The usability of V# can be massively improved, if we had C# test printer. It would be nice to have various printers sharing the common printer core: MsTest, NUnit, XUnit, etc.

Every unit test should roughly look like

[Test]
public void Test1() {
    // Allocate objects graph, including `thisObject` (if F is not static) and reference type arguments.
    try {
        TResult result = thisObject.F<T1, ..., Tn>(arg_1, ..., arg_m);
        Assert.<check result match>(...);
    } 
    catch (ExpectedException1 e) {
        // success
    }
    ....
    catch (ExpectedExceptionN e) {
        // success
    }
}

Allocating complex types

For now, classes could be allocated by using System.Runtime.Serialization.FormatterServices.GetUninitializedObject and then initializing their contents by reflection. In future, we consider to allocate objects using their public API. However, this requires additional research.

[max-arshinov]

Hi @dvvrd. I think, I have a draft of the test generator. However, complex type allocation doesn't work well for now. I am using *.vst files to generate tests; instead of

            var stats = Cover(type, outputDirectory);
            return Reproduce(stats.OutputDir);

I am running

            var stats = Cover(type, outputDirectory);
            return GenerateTests(stats.OutputDir, testFolder);

I noticed when the class I run the "Cover" method against has a constructor with parameters, VSharp just ignores it and doesn't cover this class at all.

I believe this happens in lines 881 - 903 of the Interpreter.fs file. The InitializeStatics method, precisely. Let's assume that we have the following class:

public class EmployeeService
{
    private readonly ILogger _logger;

    public EmployeeService(ILogger logger)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }
}

It would be really helpful If I could run something like:

TestGenerator.CoverAndRun<EmployeeService>(() => new EmployeeService(Mock.Of<ILogger>())); // the argument is Expression<Func<T>>

For now, I can't really move forward because I need a set of real classes to test with. I have enough real production code covered by manually-written unit tests so I can align test generation results with the real tests. The problem is that most of these classes have constructors with parameters. Please let me know if constructor support is something that can be done easily.

[dvvrd]

Hi @max-arshinov, thank you for your awesome update! I really like your proposal and I'm even more glad to hear that you are trying to work on this complex task!

TestGenerator.CoverAndRun<EmployeeService>(() => new EmployeeService(Mock.Of<ILogger>())); // the argument is Expression<Func<T>>

Good news is that we could automate this, so you don't need to write such things by hands (although I don't mind to extend API with the overload you propose, see Stage 2 description below). And this is already works, but with limitations (see below). Our plan of achieving the ideal result includes three stages.

Stage 1 (already done, see #133). I've enabled test generation for constructors (it was simply disabled). For the code that you've provided V# now generates one test passing null. However, if you add an implementation of ILogger mock like

public interface ILogger 
{
}

public class Logger : ILogger 
{
}

public class EmployeeService
{
    private readonly ILogger _logger;

    public EmployeeService(ILogger logger)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }
}

Then V# detects Logger as a suitable implementation and automatically instantiate it for passing as logger argument. You can try this right now by building the current version of constructors branch (we'll merge it into master a bit later).

Stage 2. Adding mocks (test generator-specific, we'll be glad to get some assistance on this part)

As written above, for now V# tries to detect suitable implementations for interfaces and abstract classes (and generic parameter substitutions as well) among known types and pass their instances as input parameters. If V# does not manage to find it automatically, then it just gives up on such tests.

In this situation you propose to pass mocks by hands (Mock.Of<ILogger>), but I don't see any reason not to generate them automatically. In fact, as a side-effect of symbolic execution, we have all logical specifications of type system constraints, and the only thing we need to do is to generate type an appropriate type declaration.

Of course, automated generation of such mocks heavily depends on code generator you are working on. We could discuss how to generate code for such auto-generated mocks or where to search for them automatically, and clearly you may account on us to implement all the non-trivial logic of mocks inference.

And surely, I don't mind to add the manual mocks instantiation overload, but again, we should discuss some details where to take these mocks (as code in delegate is going to be symbolically executed).

Stage 3 (more complex, requires fundamental improvement of symbolic execution engine). Reuse test coverage results for constructors to generate fancy tests of public methods of the type.

After stage 2 is accomplished, we'll generate *.vsts tests which just run constructors, like

new EmployeeService(new Logger())

As for me, these tests are ugly. A better thing to do would be to get rid of such tests as much as possible and reuse the generated instances inside tests for public functions. For example, if we add some public function into EmployeeService

public class EmployeeService
{
    private readonly ILogger _logger;

    public EmployeeService(ILogger logger)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }

    public void DoJob() {...}
}

then we would like V# to include constructor coverage into other tests whenever possible. For instance, instead of generating something like

[Test]
void Test1()
{
    Assert.Throws(new EmployeeService(null), typeof(ArgumentNullException));
}

[Test]
void Test2()
{
    new EmployeeService(Mock.Of<ILogger>());
}

[Test]
void Test3()
{
    var employeeService = instantiateEmployeeService1();
    employeeService.DoJob();
    Assert.IsTrue(...);
}

we are looking to generate just two tests with the same coverage by reusing the constructor calls:

[Test]
void Test1()
{
    Assert.Throws(new EmployeeService(null), typeof(ArgumentNullException));
}

[Test]
void Test2()
{
    var employeeService = new EmployeeService(Mock.Of<ILogger>());;
    employeeService.DoJob();
    Assert.IsTrue(...);
}

We'll elaborate on this later.

[max-arshinov]

@dvvrd thank you for the update. The "constructors" branch makes perfect sense to me. I'll check some ideas and get back to you once I have something to show.