win tests

[EgorBo]

@EgorBot help

[EgorBot]

EgorBot manual

Usage: @EgorBot [-%targets%] [other EgorBot args] [raw args for BDN] `C# snippet surrounded with triple ticks`

-%target%:       Example: "-arm" or "-intel -arm" or "-windows_x64 -linux_aws_genoa"

                 NOTE: target can be prefixed with the OS:
                  -linux_*   (implied by default)
                  -windows_*

                 Full list of targets (idiomatic names):

                  (idiomatic name)             |  Arch | OS             | Notes
                  -----------------------------|-------|----------------|------------
                  -azure_milano                |   x64 | linux          |
                  -azure_icelake               |   x64 | windows, linux |
                  -azure_ampere                | arm64 | windows, linux | Neoverse-N1
                  -azure_cobalt100             | arm64 | windows, linux | Neoverse-N2

                  -aws_sapphirelake            |   x64 | windows, linux |
                  -aws_icelake                 |   x64 | windows, linux |
                  -aws_genoa                   |   x64 | windows, linux |
                  -aws_milano                  |   x64 | windows, linux |
                  -aws_graviton2               | arm64 | linux          | Neoverse-V1
                  -aws_graviton3               | arm64 | linux          | Neoverse-V1
                  -aws_graviton4               | arm64 | linux          | Neoverse-V2

                  -gcp_emeraldrapids           |   x64 | linux          |
                  -gcp_milano,                 |   x64 | linux          |
                  -gcp_genoa,                  |   x64 | linux          |
                  -gcp_cascadeLake,            |   x64 | linux          |
                  -gcp_axion                   | arm64 | linux          | Neoverse-V2
                  -gcp_ampere                  | arm64 | linux          | Neoverse-N1

                  -hetzner_amd                 |   x64 | linux          |
                  -hetzner_milano,             |   x64 | linux          |
                  -hetzner_skylake,            |   x64 | linux          |
                  -hetzner_dedicatedmilano,    |   x64 | linux          | dedicated!
                  -hetzner_ampere,             | arm64 | linux          |

                  short aliases:
                  -arm   = -azure_cobalt100
                  -intel = -azure_cascadelake
                  -x64   = -azure_castadelake
                  -amd   = -azure_milano

-profiler:       Use 'perf record' to collect a flamegraph/hot asm. 'ultra' profiler on Windows.

-perf_events     Exact events to collect with 'perf', e.g. -perf_events ld_align_lat
                 (events are target-specific, you can find the list of the supported events in BDN_Artifacts.zip
                 when you run a benchmark with -profiler arg). Not supported for Windows.

-commit:         E.g. '-commit 72ea87f vs 380898a' or just '-commit 72ea87f'
                 Runs the benchmark on the specified commit(s) (PR is ignored)

-mono:           Use Mono runtime instead of CoreCLR for all targets. Should be possible to use
                 Mono interp too (LLVM is not supported yet).
                 Mono doesn't support -profiler (at least JIT)
                 To use mono-interp, use BDN args, e.g. --envvars MONO_ENV_OPTIONS:"--interpreter"

-nonativepgo:    Disable native PGO (Build with /p:NoPgoOptimize=true)

-pr %num%:       Run for changes in a specific PR. If bot is invoked from a PR - it takes changes
                 from that PR implicitly.

-codesafety:     Run memory safety checks for the PR, unrelated to the benchmarking.

-[args for BDN]: Args directly passed to BDN e.g. '--disasm', see
                 https://github.com/dotnet/BenchmarkDotNet/blob/master/docs/articles/guides/console-args.md

NOTE: BenchmarkRunner.Run or BenchmarkSwitcher.From* can be omitted (snippet without an entrypoint) Although, if they're presented then Program's args must be be forwarded to Run(args: args)

NOTE: Avoid [DisassemblyDiagnoser] - it causes crashes on Linux, better use --envvars DOTNET_JitDisasm:MethodName

Usage example: link

[EgorBot]

cc @EgorBo

[EgorBo]

@EgorBot -win_azure_cobalt100 -pr 106525

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

public class Bench
{
    [Benchmark]
    public string Test()
    {
        string str = "";
        for (int i = 0; i < 100; i++)
            str += i;
        return str;
    }
}

[EgorBo]

@EgorBot -win_azure_cobalt100 -profiler

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

public class Bench
{
    [Benchmark]
    public string Test()
    {
        string str = "";
        for (int i = 0; i < 100; i++)
            str += i;
        return str;
    }
}

[EgorBot]

Benchmark results on windows_azure_cobalt100

BenchmarkDotNet v0.14.0, Windows 11 (10.0.26100.2314)
azure_cobalt100
DefaultJob : .NET 9.0.0 (9.0.24.52809), Arm64 RyuJIT AdvSIMD
StdDev=0.0175 μs

Method	Mean	Error
Test	1.754 μs	0.0187 μs

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt)

[EgorBo]

@EgorBot -win_azure_cobalt100 -profiler

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

public class Bench
{
    [Benchmark]
    public string Test()
    {
        string str = "";
        for (int i = 0; i < 100; i++)
            str += i;
        return str;
    }
}

[EgorBot]

❌ Failed on AzureCobalt100: Job failed, see logs.

cc @EgorBo (logs)

[EgorBot]

Benchmark results on windows_azure_cobalt100

BenchmarkDotNet v0.14.0, Windows 11 (10.0.26100.2314)
azure_cobalt100
DefaultJob : .NET 9.0.0 (9.0.24.52809), Arm64 RyuJIT AdvSIMD
StdDev=0.0159 μs

Method	Mean	Error
Test	1.773 μs	0.0170 μs

BDN_Artifacts.zip

Profiling for first [Benchmark]

• Ultra profiler: link

cc @EgorBo (agent_logs.txt)

[EgorBo]

@EgorBot -win_azure_cobalt100 -pr 106525

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

public class Bench
{
    [Benchmark]
    public string Test()
    {
        string str = "";
        for (int i = 0; i < 100; i++)
            str += i;
        return str;
    }
}

[EgorBot]

Benchmark results on windows_azure_cobalt100

BenchmarkDotNet v0.14.0, Windows 11 (10.0.26100.2314)
azure_cobalt100
  Job-HIWTCY : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
  Job-UBBDLI : .NET 9.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
RatioSD=0.02

Method	Job	Toolchain	Mean	Error	Ratio
Test	Job-HIWTCY	\core_root_base\corerun.exe	2.009 μs	0.0259 μs	1.00
Test	Job-UBBDLI	\core_root_diff\corerun.exe	1.924 μs	0.0237 μs	0.96

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt)

[EgorBo]

@EgorBot -win_azure_cobalt100 -pr 106525 -profiler

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

public class Bench
{
    [Benchmark]
    public string Test()
    {
        string str = "";
        for (int i = 0; i < 100; i++)
            str += i;
        return str;
    }
}

[EgorBo]

@EgorBot -intel

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

public class Bench
{
    [Benchmark]
    public string Test()
    {
        string str = "";
        for (int i = 0; i < 100; i++)
            str += i;
        return str;
    }
}

[EgorBot]

Benchmark results on linux_azure_cascadelake

BenchmarkDotNet v0.14.0, Ubuntu 24.04.1 LTS (Noble Numbat)
Intel Xeon Platinum 8370C CPU 2.80GHz, 1 CPU, 4 logical and 2 physical cores
DefaultJob : .NET 9.0.0 (9.0.24.52809), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
StdDev=0.0232 μs

Method	Mean	Error
Test	2.357 μs	0.0262 μs

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt)

[EgorBot]

Benchmark results on windows_azure_cobalt100

BenchmarkDotNet v0.14.0, Windows 11 (10.0.26100.2314)
azure_cobalt100
  Job-FTTWUS : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
  Job-ZHBMYD : .NET 9.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD

Method	Job	Toolchain	Mean	Error	Ratio
Test	Job-FTTWUS	\core_root_base\corerun.exe	1.822 μs	0.0109 μs	1.00
Test	Job-ZHBMYD	\core_root_diff\corerun.exe	1.765 μs	0.0065 μs	0.97

BDN_Artifacts.zip

Profiling for first [Benchmark]

• Ultra profiler: Main vs PR

cc @EgorBo (agent_logs.txt)