Is it possible to benchmark published ASP.NET Core web applications?

[PonchoPowers]

Is it possible to benchmark published ASP.NET Core web applications?

Eilon from MS mentioned to me that: "Testing a not-published app could yield very different perf results than a published app. There are a variety of things that can happen as part of the publish process, such as compiling Razor views, which would greatly affect results."

Related: https://github.com/aspnet/Home/issues/2897#issuecomment-367856935

This is obviously quite a big deal if so.

[PonchoPowers]

So this is what I did, I published the web application and removed the reference to the project and instead referenced the published dll.

When I try to build and run in Visual Studio, or the command line, I get the following error: "System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Configuration.UserSecrets, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified."

But if I reference the project again everything runs.

Is there a solution to this?

[adamsitnik]

@Matthew-Bonner could you provide some more context? What are you trying to measure? If I know the problem I can provide better help

[PonchoPowers]

Firstly, I would like to measure the site in it's published state based on the comments about published sites running differently to built sites as linked to in the opening comment.

I am aware of the facility to test URL's using RunUrl, but I want to test more granular than a full request, such as specific method calls that interact with other systems, so RunUrl won't work for me.

We have internal API's that we call and I'd like to use BenchmarkDotNet to prove that it is not the code in our application that is responsible for the slow running of certain aspects of our web application.

The idea is to use BenchmarkDotNet to make the same calls to the same methods over and over again as it does to put a bit of stress on the API that is being used, to simulate live traffic.

Now, I'm not sure if BenchmarkDotNet was designed to do this, but I'd like to use it for this anyway, and I'd like to use it against our published website so that there is no comeback that the results might be skewed because we didn't test against our published web application.

[adamsitnik]

We have internal API's that we call and I'd like to use BenchmarkDotNet to prove that it is not the code in our application that is responsible for the slow running of certain aspects of our web application.

When investigating performance problems you need to remember that benchmark can tell only how fast given method runs. Not why is it slow or which part of the method takes most of the time.

Example: If method A calls methods B and C, the benchmark is going to tell you how long does it take to run A. Profiler is also going to tell you what part of it took B and C.

I believe that you should use a profiler. What I would do if I were you:

1. Publish the app exactly the same way you publish it when releasing to PROD
2. Use some software for web apps stress testing to generate traffic.
3. Attach a profiler to your app when it's under a stress test. Collect data for 10-30 seconds and detach the profiler.
4. Use the profiler to find out what takes most of the time.

One of the profilers that you can use is PerfView. It is a very powerful tool, however, the UX is not a strong part of it. Here you can watch some tutorial about it.

More user-friendly profiler is dotTrace from JetBrains. It's not for free, but you can check the trial version.

If you have access to Pluralsight then you might also want to see this video tutorial about measuring .NET Performance.

@Matthew-Bonner please let me know if it helps or not.

[PonchoPowers]

We have a profiler but the profiler does not allow us to profile during the build process.

The idea of using BenchmarkDotNet is to benchmark methods overnight as part of the build process. It is pointless to benchmark unit tests so we are benchmarking real world scenarios. But as part of that we also want to run some more narrowly scoped tests in more critical areas.

I can see you put a lot of effort into your suggestion but it does not help unfortunately.

Presumably by your answers, it is not possible to benchmark published ASP.NET Core web apps?

Can I just check you understand what I mean by this?

[adamsitnik]

I am doing performance investigations for a living and I always try to understand what is the problem. What the customer needs, not what the customer wants. These are usually two different things. Now I know what is your problem. I asked the question because many, many people use BenchmarkDotNet and don't even know about the existence of profilers. I try to guide them if I can.

BenchmarkDotNet uses dotnet cli to build the apps, but we call dotnet resore, dotnet build and dotnet $app.dll. We don't publish the apps. If you want to publish the app, you need to implement a custom toolchain. We have one toolchain which is using dotnet publish, but it's publishing self-contained apps for custom CoreCLR/FX scenario. Code. So you would need to copy the code and modify it.

If you want to measure the performance of a published ASP.NET app by making some web requests you need BenchmarkDotNet to publish the app and invoke the desired controller actions.

You can use [GlobalSetup] to mark a method which does the publish. It's going to be invoked once. The method should also start the web server. Then your benchmark needs to make some web requests.

Some pseudo code:

public class WebServerRequestBenchmark
{
    private readonly HttpClient httpClient = new HttpClient();

    private Process host;

    [GlobalSetup]
    public void Setup()
    {
        CommandExecutor.ExecuteCommand("dotnet", arguments: $"publish -c Release -f {TargetFrameworkMoniker} {Path.GetFileName(projectPath)}");

        var exePath = Path.Combine(Path.GetDirectoryName(projectPath), "bin", "Release", TargetFrameworkMoniker, "publish",
        $"{Path.GetFileName(Path.GetFileNameWithoutExtension(projectPath))}.dll");

        host = CommandExecutor.Start("dotnet", arguments: Path.GetFileName(exePath),  workingDirectory: Path.GetDirectoryName(exePath));

        string line = null;
        while ((line = host.StandardOutput.ReadLine()) != null)
        {
            if (line.Contains("Now listening on:"))
                hostUrl = line.Substring(line.IndexOf("http")).Trim();

            if (line.Contains("Application started.")) // it means the app is ready to use
                break;

            if (host.HasExited)
                break;
        }
    }

    [Benchmark]
    public HttpStatusCode InvokeController() => httpClient.GetAsync($"{hostUrl}/Action").GetAwaiter().GetResult().StatusCode;

    [GlobalCleanup]
    public void CloseHost()
    {
        host.Kill();
        host.Dispose();

        httpClient.Dispose();
    }
}

[PonchoPowers]

Sorry, I know from experience that I'm pretty bad at explaining things, so I was just checking to see if you were following what I was going on about as I sometimes edit what I say two or three times before I think it makes any sense :)

I will give your solution a try and see how I get on, thanks for assisting me with this.

[PonchoPowers]

Can I ask, as part of the work I'm doing in relation to this issue, that the ConsoleHandler, SynchronousProcessOutputLoggerWithDiagnoser, DotNetCliBuilder.RestoreCommand, DotNetCliBuilder.GetBuildCommand and ProcessExtensions.SetEnvironmentVariables is made public rather than internal.

I'm having to implement my own DotNetCliExecutor so that I can modify the BuildArgs method and as part of doing so I'm trying to keep my code aligned to this codebase as much as possible.

I also feel it is important for others wanting to implement their own custom toolchain to have access to these classes.

[adamsitnik]

@Matthew-Bonner please make the required types public and send a PR. I am going to merge it asap

[PonchoPowers]

Hi Adam, will do, just working out what changes I need exactly then will create a PR request.

[PonchoPowers]

PR: https://github.com/dotnet/BenchmarkDotNet/pull/670

Not as many changes were required in the end to get this working.

I created my own generator, builder and toolchain class.

[PonchoPowers]

Here is my code in case you are interested in what I did:

    public class AspNetCoreProjBuilder : DotNetCliBuilder
    {
        public override string RestoreCommand => "restore --no-dependencies";

        public override string GetBuildCommand(string frameworkMoniker, bool justTheProjectItself, string configuration)
            => $"publish --framework {frameworkMoniker} --configuration {configuration} --no-restore"
               + (justTheProjectItself ? " --no-dependencies" : string.Empty);

        public AspNetCoreProjBuilder(string targetFrameworkMoniker, string customDotNetCliPath)
            : base(targetFrameworkMoniker, customDotNetCliPath)
        {
        }
    }

public class AspNetCoreProjGenerator : CsProjGenerator
    {
        public AspNetCoreProjGenerator(string targetFrameworkMoniker, Func<Platform, string> platformProvider, string runtimeFrameworkVersion = null)
            : base(targetFrameworkMoniker, platformProvider, runtimeFrameworkVersion)
        {
        }

        protected override string GetBinariesDirectoryPath(string buildArtifactsDirectoryPath, string configuration)
        {
            return Path.Combine(buildArtifactsDirectoryPath, "bin", configuration, TargetFrameworkMoniker, "publish");
        }
    }

public class AspNetCoreToolchain : Toolchain
    {
        public static readonly IToolchain AspNetCoreApp11 = From(NetCoreAppSettings.NetCoreApp11);
        public static readonly IToolchain AspNetCoreApp12 = From(NetCoreAppSettings.NetCoreApp12);
        public static readonly IToolchain AspNetCoreApp20 = From(NetCoreAppSettings.NetCoreApp20);
        public static readonly IToolchain AspNetCoreApp21 = From(NetCoreAppSettings.NetCoreApp21);

        public static readonly IToolchain Current = From(AspNetCoreAppVersion.GetCurrentVersion());

        private AspNetCoreToolchain(string name, IGenerator generator, IBuilder builder, IExecutor executor, string customDotNetCliPath)
            : base(name, generator, builder, executor)
        {
            CustomDotNetCliPath = customDotNetCliPath;
        }

        private string CustomDotNetCliPath { get; }

        public static IToolchain From(NetCoreAppSettings settings)
            => new AspNetCoreToolchain(settings.Name,
                new AspNetCoreProjGenerator(settings.TargetFrameworkMoniker, PlatformProvider, settings.RuntimeFrameworkVersion),
                new AspNetCoreProjBuilder(settings.TargetFrameworkMoniker, settings.CustomDotNetCliPath),
                new DotNetCliExecutor(settings.CustomDotNetCliPath),
                settings.CustomDotNetCliPath);

        private static string PlatformProvider(Platform platform) => platform.ToConfig();

        public override bool IsSupported(BenchmarkDotNet.Running.Benchmark benchmark, ILogger logger, IResolver resolver)
        {
            if (!base.IsSupported(benchmark, logger, resolver))
            {
                return false;
            }

            if (string.IsNullOrEmpty(CustomDotNetCliPath) && string.IsNullOrEmpty(HostEnvironmentInfo.GetCurrent().DotNetSdkVersion.Value))
            {
                logger.WriteLineError($"BenchmarkDotNet requires dotnet cli toolchain to be installed, benchmark '{benchmark.DisplayInfo}' will not be executed");
                return false;
            }

            if (!string.IsNullOrEmpty(CustomDotNetCliPath) && !File.Exists(CustomDotNetCliPath))
            {
                logger.WriteLineError($"Povided custom dotnet cli path does not exist, benchmark '{benchmark.DisplayInfo}' will not be executed");
                return false;
            }

            if (benchmark.Job.HasValue(EnvMode.JitCharacteristic) && benchmark.Job.ResolveValue(EnvMode.JitCharacteristic, resolver) == Jit.LegacyJit)
            {
                logger.WriteLineError($"Currently dotnet cli toolchain supports only RyuJit, benchmark '{benchmark.DisplayInfo}' will not be executed");
                return false;
            }
            if (benchmark.Job.ResolveValue(GcMode.CpuGroupsCharacteristic, resolver))
            {
                logger.WriteLineError($"Currently project.json does not support CpuGroups (app.config does), benchmark '{benchmark.DisplayInfo}' will not be executed");
                return false;
            }
            if (benchmark.Job.ResolveValue(GcMode.AllowVeryLargeObjectsCharacteristic, resolver))
            {
                logger.WriteLineError($"Currently project.json does not support gcAllowVeryLargeObjects (app.config does), benchmark '{benchmark.DisplayInfo}' will not be executed");
                return false;
            }

            return true;
        }
    }