Enabling AllowVeryLargeObjects has no effect on Framework benchmark process with Core host process

[LanceUMatthews]

I am using v0.12.1 in a console project to run benchmarks on both .NET Core and Framework. I have configured the .csproj as follows...

<TargetFrameworks>netcoreapp3.1;net48</TargetFrameworks>

...and then launch it with...

dotnet run --configuration release --framework netcoreapp3.1 benchmark

This works fine. (That trailing benchmark parameter is handled by my code to then call BenchmarkDotNet.Running.BenchmarkRunner.Run<Benchmarks>();)

After increasing a [Params()] value that controls the Capacity of a List<>, I started getting failures in the Framework benchmarks...

OutOfMemoryException! BenchmarkDotNet continues to run additional iterations until desired accuracy level is achieved. It's possible only if the benchmark method doesn't have any side-effects. If your benchmark allocates memory and keeps it alive, you are creating a memory leak. You should redesign your benchmark and remove the side-effects. You can use OperationsPerInvoke, IterationSetup and IterationCleanup to do that.

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Array dimensions exceeded supported range.
   at System.Collections.Generic.List`1..ctor(Int32 capacity)

To remedy this, I changed my configuration from...

[SimpleJob(RuntimeMoniker.NetCoreApp31)]
[SimpleJob(RuntimeMoniker.Net48)]
public class Benchmarks
{

...to...

Job baseJob = Job.Dry;
IConfig config = DefaultConfig.Instance
    .AddJob(baseJob.WithRuntime(ClrRuntime.Net48).WithGcAllowVeryLargeObjects(true))
    .AddJob(baseJob.WithRuntime(CoreRuntime.Core31));

BenchmarkDotNet.Running.BenchmarkRunner.Run<Benchmarks>(config);

What I found, though, is that the Framework benchmarks all still failed with that same error when passing --framework netcoreapp3.1 to dotnet run. Immediately before the error the log even shows that AllowVeryLargeObjects is enabled...

// BeforeAnythingElse

// Benchmark Process Environment Information: // Runtime=.NET Framework 4.8 (4.8.4200.0), X64 RyuJIT // GC=Concurrent Workstation // Job: Dry(AllowVeryLargeObjects=True, IterationCount=1, LaunchCount=1, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=1)

...but only after changing to --framework net48 would the benchmarks succeed.

My question is, is this expected behavior and, if so, can you help me understand why that is? I've read Toolchains, among other articles, but I'm still not seeing how the runtime of the host process would affect a runtime-specific benchmark process in this way.

I should add that my program also takes a test command line parameter, and my project has an App.config file configured so that code path, too, can successfully create very large arrays. With that file in place and running BenchmarkDotNet with --framework netcoreapp3.1 and KeepBenchmarkFiles(true) added to config above, I see that same <gcAllowVeryLargeObjects enabled="true" /> directive does get propagated to...

• bin\release\net48\MyProject.exe.config,
• bin\release\netcoreapp3.1\Dry\bin\Release\net48\Dry.exe.config,
• bin\release\netcoreapp3.1\Dry\bin\Release\netcoreapp3.1\MyProject.dll.config, and
• bin\release\netcoreapp3.1\MyProject.dll.config

[adamsitnik]

Hi @LanceUMatthews

I see that same directive does get propagated to.

Thanks for digging that deep! Does it also get propagated to: bin\release\netcoreapp3.1\Dry\bin\Release\netcoreapp3.1\Dry.dll.config ? (MyProject -> Dry)

the logic for copying config files from the project that defines benchmark lives here:

https://github.com/dotnet/BenchmarkDotNet/blob/09288954791df64a3da162df705b7701e1c7e8b9/src/BenchmarkDotNet/Toolchains/GeneratorBase.cs#L96-L105

My question is, is this expected behavior and, if so, can you help me understand why that is?

It's not expected and it's most probably caused by differences in toolchains. When you run dotnet run -f net48 the RoslynToolchain is used to generate and build the boilerplate project.

When you use dotnet run -f netcoreapp3.1 to run net48 benchmarks, the CsProjClassicNetToolchain is used

There is most probably a bug somewhere in one of them. But... according to https://docs.microsoft.com/en-us/dotnet/core/run-time-config/garbage-collector#allow-large-objects the only way to enable Large Objects in .NET Core is via env vars.

That trailing benchmark parameter is handled by my code to then call

In case you have implemented filtering on your own, please do know that this is supported by BenchmarkSwitcher and the recommended way to run the benchmarks for multiple runtimes is by using BenchmarkSwitcher and official console line arguments described here: https://benchmarkdotnet.org/articles/configs/toolchains.html#multiple-frameworks-support

So if you switch to BenchmarkSwitcher you should be able to achieve what you want by doing:

dotnet run -c Release -f net48 --filter $TypeName --runtimes net48 netcoreapp3.1 --enVars COMPlus_gcAllowVeryLargeObjects:1

Also, are you sure that your benchmark has no side effects https://github.com/dotnet/performance/blob/master/docs/microbenchmark-design-guidelines.md#No-Side-Effects and you are going to use Very Large Objects in Production and you care about this scenario?

[LanceUMatthews]

Hi, @adamsitnik . Thanks for your response.

Does it also get propagated to: bin\release\netcoreapp3.1\Dry\bin\Release\netcoreapp3.1\Dry.dll.config ? (MyProject -> Dry)

No, that Dry.dll.config file contains no configuration...

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <runtime>
  </runtime>
</configuration>

It's not expected and it's most probably caused by differences in toolchains.

I see. So the host process determines which toolchain is used to generate the benchmark project/process.

There is most probably a bug somewhere in one of them. But... according to https://docs.microsoft.com/en-us/dotnet/core/run-time-config/garbage-collector#allow-large-objects the only way to enable Large Objects in .NET Core is via env vars.

Based on this my understanding is gcAllowVeryLargeObjects is enabled by default in .NET Core.

Related to that, I originally tried creating my configuration like this...

IConfig config = DefaultConfig.Instance;

foreach (Runtime runtime in new Runtime[] { ClrRuntime.Net48, CoreRuntime.Core31 })
{
    config = config.AddJob(
        Job.Dry.WithRuntime(runtime).WithGcAllowVeryLargeObjects(true)
    );
}

...so both jobs have AllowVeryLargeObjects enabled, but that resulted in this error...

Currently project.json does not support gcAllowVeryLargeObjects (app.config does), benchmark 'Benchmarks.MyBenchmarkMethod: Dry(Runtime=.NET Core 3.1, AllowVeryLargeObjects=True, IterationCount=1, LaunchCount=1, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=1) [DataListSize=1000000000]' will not be executed

My expectation was that WithGcAllowVeryLargeObjects(true)/new GcMode() { AllowVeryLargeObjects = true } meant "Do whatever you have to do, if anything, to enable large object support", but it turned out to be "Definitely create this configuration to enable large object support". So, I had to be careful about on which job I explicitly request large object support even though both would be using it. Although, since both members are clearly named after the backing configuration element I can see how they're so strongly tied to it.

In case you have implemented filtering on your own, please do know that this is supported by BenchmarkSwitcher and the recommended way to run the benchmarks for multiple runtimes is by using BenchmarkSwitcher and official console line arguments described here: https://benchmarkdotnet.org/articles/configs/toolchains.html#multiple-frameworks-support

I had not come across BenchmarkSwitcher until this latest trek through the documentation, but it looks like something I should become familiar with, particularly for the filtering capability.

My benchmark parameter is just calling BenchmarkRunner.Run<Benchmarks>(), and the test parameter is calling those same benchmark methods and outputting the return values so the user can see they're correct. (This little project is all for a Stack Overflow answer, so simplicity and easy-to-invoke code with easy-to-digest output is key.) I imagine I can still use BenchmarkSwitcher as long as I strip my parameter off first.

Also, are you sure that your benchmark has no side effects https://github.com/dotnet/performance/blob/master/docs/microbenchmark-design-guidelines.md#No-Side-Effects and you are going to use Very Large Objects in Production and you care about this scenario?

Yes, very large List<>s is exactly the scenario in question and I believe there are no side-effects. Here's how I create my List<>...

[Params(1_000_000_000)]
public int DataListSize
{
    get; set;
}

[GlobalSetup()]
public void GlobalSetup()
{
    DataList = new List<int>(DataListSize);
}

The benchmark methods don't do anything to change the Capacity of the List<> but they could modify or remove its elements, so here's how I set them back to their initial state...

[IterationSetup()]
public void IterationSetup()
{
    for (int i = 0; i < DataListSize; i++)
        DataList.Add(i);
}

[IterationCleanup()]
public void IterationCleanup()
{
    DataList.Clear();
}

I also have...

[GlobalCleanup()]
public void GlobalCleanup()
{
    int generation = GC.GetGeneration(DataList);

    DataList = null;
    GC.Collect(generation, GCCollectionMode.Forced, true);
}

...but, now that I know that GcMode.Force (not only exists but) is true by default, I take it that is redundant?

[LanceUMatthews]

I can confirm that...

if (string.Equals(args[0], "benchmark", StringComparison.OrdinalIgnoreCase))
{
    string[] switcherArgs = args.Skip(1).ToArray();

    BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(switcherArgs);
}

...works when run with...

dotnet run --configuration release --framework net48 -- benchmark --filter MyProject.Benchmarks.* --job Dry --runtimes netcoreapp3.1 net48

...as long as my benchmarks don't use very large objects (and I stick with --job Dry, though that is another matter: #1521). To get it to work with large objects I tried...

IConfig config = DefaultConfig.Instance.AddJob(
    Job.Default.WithGcAllowVeryLargeObjects(true).AsDefault()
);

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(switcherArgs, config);

...but then I get the error about not being able to add gcAllowVeryLargeObjects to the .NET Core project's configuration. Changing that to...

IConfig config = DefaultConfig.Instance.AddJob(
    Job.Clr.WithGcAllowVeryLargeObjects(true).AsDefault()
);

...won't compile because Job.Clr is marked [Obsolete("...", true)]. This does compile...

IConfig config = DefaultConfig.Instance.AddJob(
    Job.Default.WithRuntime(ClrRuntime.Net48).WithGcAllowVeryLargeObjects(true).AsDefault()
);

...but that's referencing a specific version and I get the worst of both worlds, anyways: the .NET Core project fails due to (trying to) enabling gcAllowVeryLargeObjects, and the .NET Framework project fails due to (evidently) not enabling gcAllowVeryLargeObjects...

// Execute: ...\MyProject\bin\release\net48\3a16e50c-b648-4e11-af5a-356496b6cd87\bin\Release\net48\3a16e50c-b648-4e11-af5a-356496b6cd87.exe --benchmarkName "MyProject.Benchmarks.MyBenchmarkMethod(DataListSize: 1000000000)" --job "Runtime=.NET 4.8, AllowVeryLargeObjects=True, Toolchain=net48, InvocationCount=1, UnrollFactor=1" --benchmarkId 0 in // BeforeAnythingElse

// Benchmark Process Environment Information: // Runtime=.NET Framework 4.8 (4.8.4200.0), X64 RyuJIT // GC=Concurrent Workstation // Job: Job-WJQJHA(AllowVeryLargeObjects=True, InvocationCount=1, UnrollFactor=1)

OutOfMemoryException! BenchmarkDotNet continues to run additional iterations until desired accuracy level is achieved. It's possible only if the benchmark method doesn't have any side-effects. If your benchmark allocates memory and keeps it alive, you are creating a memory leak. You should redesign your benchmark and remove the side-effects. You can use OperationsPerInvoke, IterationSetup and IterationCleanup to do that.

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Array dimensions exceeded supported range.
   at System.Collections.Generic.List`1..ctor(Int32 capacity)

Given the current incompatibility between WithGcAllowVeryLargeObjects(true) and Core jobs, is there any way that I can make enabling that the default only for CLR jobs?

[adamsitnik]

I've sent a fix #1519

[pmaytak]

@adamsitnik Is the fix for this planned to be included in some future release? Meanwhile is there a workaround to enable large objects? Thank you.