thomhurst / TUnit

A modern, fast and flexible .NET testing framework
MIT License
1.02k stars 8 forks source link

TUnit

A modern, flexible and fast testing framework for .NET 8 and up. With Native AOT and Trimmed Single File application support included!

nuget Nuget GitHub Workflow Status (with event) GitHub last commit (branch) License

Documentation

See here: https://thomhurst.github.io/TUnit/

IDE

TUnit is built on top of the newer Microsoft.Testing.Platform, as opposed to the older VSTest platform. Because the infrastructure behind the scenes is new and different, you may need to enable some settings. This should just be a one time thing.

Visual Studio

Visual Studio is supported on the Preview version currently.

Rider

Rider is supported. The Enable Testing Platform support option must be selected in Settings > Build, Execution, Deployment > Unit Testing > VSTest.

VS Code

Visual Studio Code is supported.

CLI

dotnet CLI - Fully supported. Tests should be runnable with dotnet test, dotnet run, dotnet exec or executing an executable directly. See the docs for more information!

Features

Installation

dotnet add package TUnit --prerelease

Example test

    private static readonly TimeOnly Midnight = TimeOnly.FromTimeSpan(TimeSpan.Zero);
    private static readonly TimeOnly Noon = TimeOnly.FromTimeSpan(TimeSpan.FromHours(12));

    [Test]
    public async Task IsMorning()
    {
        var time = GetTime();

        await Assert.That(time).IsAfterOrEqualTo(Midnight)
            .And.IsBefore(Noon);
    }

or with more complex test orchestration needs

    [Before(Class)]
    public static async Task ClearDatabase(ClassHookContext context) { ... }

    [After(Class)]
    public static async Task AssertDatabaseIsAsExpected(ClassHookContext context) { ... }

    [Before(Test)]
    public async Task CreatePlaywrightBrowser(TestContext context) { ... }

    [After(Test)]
    public async Task DisposePlaywrightBrowser(TestContext context) { ... }

    [Retry(3)]
    [Test, DisplayName("Register an account")]
    [MethodData(nameof(GetAuthDetails))]
    public async Task Register(string username, string password) { ... }

    [Repeat(5)]
    [Test, DependsOn(nameof(Register))]
    [MethodData(nameof(GetAuthDetails))]
    public async Task Login(string username, string password) { ... }

    [Test, DependsOn(nameof(Login), [typeof(string), typeof(string)])]
    [MethodData(nameof(GetAuthDetails))]
    public async Task DeleteAccount(string username, string password) { ... }

    [Category("Downloads")]
    [Timeout(300_000)]
    [Test, NotInParallel(Order = 1)]
    public async Task DownloadFile1() { ... }

    [Category("Downloads")]
    [Timeout(300_000)]
    [Test, NotInParallel(Order = 2)]
    public async Task DownloadFile2() { ... }

    [Repeat(10)]
    [Test]
    [Arguments(1)]
    [Arguments(2)]
    [Arguments(3)]
    [DisplayName("Go to the page numbered $page")]
    public async Task GoToPage(int page) { ... }

    [Category("Cookies")]
    [Test, Skip("Not yet built!")]
    public async Task CheckCookies() { ... }

    [Test, Explicit, WindowsOnlyTest, RetryHttpServiceUnavailable(5)]
    [Property("Some Key", "Some Value")]
    public async Task Ping() { ... }

    [Test]
    [ParallelLimit<LoadTestParallelLimit>]
    [Repeat(1000)]
    public async Task LoadHomepage() { ... }

    public static IEnumerable<(string Username, string Password)> GetAuthDetails()
    {
        yield return ("user1", "password1");
        yield return ("user2", "password2");
        yield return ("user3", "password3");
    }

    public class WindowsOnlyTestAttribute : SkipAttribute
    {
        public WindowsOnlyTestAttribute() : base("Windows only test")
        {
        }

        public override Task<bool> ShouldSkip(TestContext testContext)
        {
            return Task.FromResult(!OperatingSystem.IsWindows());
        }
    }

    public class RetryHttpServiceUnavailableAttribute : RetryAttribute
    {
        public RetryHttpServiceUnavailableAttribute(int times) : base(times)
        {
        }

        public override Task<bool> ShouldRetry(TestInformation testInformation, Exception exception, int currentRetryCount)
        {
            return Task.FromResult(exception is HttpRequestException { StatusCode: HttpStatusCode.ServiceUnavailable });
        }
    }

    public class LoadTestParallelLimit : IParallelLimit
    {
        public int Limit => 50;
    }

Motivations

TUnit is inspired by NUnit and xUnit - two of the most popular testing frameworks for .NET.

It aims to build upon the useful features of both while trying to address any pain points that they may have.

Read more here

Benchmark

Scenario: Building the test project

ubuntu-latest


BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  DefaultJob : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
Method Mean Error StdDev
Build_TUnit 1.585 s 0.0294 s 0.0260 s
Build_NUnit 1.466 s 0.0250 s 0.0234 s
Build_xUnit 1.521 s 0.0268 s 0.0251 s
Build_MSTest 1.508 s 0.0294 s 0.0315 s

windows-latest


BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  DefaultJob : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
Method Mean Error StdDev
Build_TUnit 1.496 s 0.0290 s 0.0310 s
Build_NUnit 1.354 s 0.0265 s 0.0248 s
Build_xUnit 1.354 s 0.0185 s 0.0164 s
Build_MSTest 1.371 s 0.0228 s 0.0214 s

macos-latest


BenchmarkDotNet v0.14.0, macOS Sonoma 14.6.1 (23G93) [Darwin 23.6.0]
Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), Arm64 RyuJIT AdvSIMD
  DefaultJob : .NET 8.0.8 (8.0.824.36612), Arm64 RyuJIT AdvSIMD
Method Mean Error StdDev Median
Build_TUnit 888.7 ms 25.57 ms 72.53 ms 866.6 ms
Build_NUnit 769.6 ms 15.24 ms 37.39 ms 760.5 ms
Build_xUnit 757.7 ms 14.96 ms 23.73 ms 753.7 ms
Build_MSTest 819.9 ms 15.95 ms 18.37 ms 821.8 ms

Scenario: A single test that completes instantly (including spawning a new process and initialising the test framework)

macos-latest


BenchmarkDotNet v0.14.0, macOS Sonoma 14.6.1 (23G93) [Darwin 23.6.0]
Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), Arm64 RyuJIT AdvSIMD
  DefaultJob : .NET 8.0.8 (8.0.824.36612), Arm64 RyuJIT AdvSIMD
Method Mean Error StdDev Median
TUnit_AOT 82.52 ms 2.254 ms 6.284 ms 79.87 ms
TUnit 427.05 ms 8.103 ms 10.248 ms 428.30 ms
NUnit 707.21 ms 14.111 ms 13.199 ms 710.59 ms
xUnit 728.39 ms 14.527 ms 24.668 ms 726.60 ms
MSTest 661.63 ms 13.084 ms 17.013 ms 661.41 ms

ubuntu-latest


BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  DefaultJob : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
Method Mean Error StdDev
TUnit_AOT 45.79 ms 1.034 ms 3.048 ms
TUnit 769.56 ms 15.116 ms 19.117 ms
NUnit 1,359.34 ms 19.121 ms 16.950 ms
xUnit 1,345.91 ms 20.291 ms 18.980 ms
MSTest 1,200.94 ms 8.903 ms 8.328 ms

windows-latest


BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  DefaultJob : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
Method Mean Error StdDev Median
TUnit_AOT 79.25 ms 1.559 ms 1.796 ms 78.05 ms
TUnit 777.50 ms 15.522 ms 20.183 ms 777.96 ms
NUnit 1,338.90 ms 12.597 ms 11.784 ms 1,340.02 ms
xUnit 1,304.54 ms 8.283 ms 7.748 ms 1,304.07 ms
MSTest 1,178.41 ms 20.441 ms 19.120 ms 1,171.72 ms

Scenario: A test that takes 50ms to execute, repeated 100 times (including spawning a new process and initialising the test framework)

ubuntu-latest


BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  DefaultJob : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
Method Mean Error StdDev
TUnit_AOT 81.13 ms 1.510 ms 3.408 ms
TUnit 837.17 ms 16.424 ms 22.481 ms
NUnit 6,339.53 ms 24.973 ms 23.360 ms
xUnit 6,389.66 ms 25.984 ms 24.305 ms
MSTest 6,315.17 ms 23.841 ms 22.301 ms

windows-latest


BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
  DefaultJob : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2
Method Mean Error StdDev
TUnit_AOT 132.1 ms 2.62 ms 6.71 ms
TUnit 836.6 ms 15.01 ms 21.53 ms
NUnit 7,531.8 ms 33.75 ms 31.57 ms
xUnit 7,533.1 ms 29.32 ms 27.43 ms
MSTest 7,455.9 ms 21.09 ms 19.72 ms

macos-latest


BenchmarkDotNet v0.14.0, macOS Sonoma 14.6.1 (23G93) [Darwin 23.6.0]
Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores
.NET SDK 8.0.401
  [Host]     : .NET 8.0.8 (8.0.824.36612), Arm64 RyuJIT AdvSIMD
  DefaultJob : .NET 8.0.8 (8.0.824.36612), Arm64 RyuJIT AdvSIMD
Method Mean Error StdDev
TUnit_AOT 242.2 ms 11.27 ms 33.24 ms
TUnit 558.0 ms 22.70 ms 66.92 ms
NUnit 14,194.2 ms 283.06 ms 558.72 ms
xUnit 14,418.4 ms 287.87 ms 607.21 ms
MSTest 14,295.4 ms 283.16 ms 565.49 ms