Unexpected Runs of SetupMethods

[swtrse]

After getting deeper into benchmarks and attempts to speed up certain tests I run into an unexpected behavior. I was wondering if that is intentional or I found a bug in the system.

My benchmark is designed like this

[ShortRunJob(RuntimeMoniker.Net80)]
[ShortRunJob(RuntimeMoniker.Net90)]
[GenericTypeArguments(typeof(int))]
[GenericTypeArguments(typeof(long))]
public class MyBenchmark<T> where T : notnull
{
    [Params(0,1,2)] public int MyProperty1 { get; [UsedImplicitly] set; }
    [Params(0,1,2)] public int MyProperty2 { get; [UsedImplicitly] set; }

   [GlobalSetup]
    public void A_Global_Setup()
    {
        // Very time consuming preparations....
    }

    [IterationSetup(Target = "Benchmark1")]
    public void A_Iteration_Setup_Benchmark1()
    {
        // Preparations needed for each testrun....
    }

   [IterationSetup(Target = "Benchmark2")]
    public void A_Iteration_Setup_Benchmark2()
    {
        // Preparations needed for each testrun....
    }

    [Benchmark]
    public void Benchmark1()
    {
        // Do Stuff
    }

    [Benchmark]
    public void Benchmark2()
    {
        // Do Stuff
    }
}

Now what I see in the Logs is that the Test runs like

• Benchmark Settings (Runtime =.NET 8.0, MyProperty1 = 0, MyProperty2 = 0, T = int)
  • A_Global_Setup
  • A_Iteration_Setup_Benchmark1 (Multiple times...)
  • Benchmark1()
  • A_Global_Setup
  • A_Iteration_Setup_Benchmark2 (Multiple times...)
  • Benchmark2()
• Benchmark Settings (Runtime =.NET 8.0, MyProperty1 = 0, MyProperty2 = 0, T = long)
  • A_Global_Setup
  • A_Iteration_Setup_Benchmark1 (Multiple times...)
  • Benchmark1() (Multiple times...)
  • A_Global_Setup
  • A_Iteration_Setup_Benchmark2 (Multiple times...)
  • Benchmark2() (Multiple times...) ... and so on

I do get that Iteration Setups are running multiple times for Overhead detection. Prepare the prerequisites for each call of Benchmark1() and so on. That is ok. What looks like an error to me is that the Global Setup is running every time too. Even when the Benchmark Settings did not change. I noticed this because I wanted to do some heavy lifting in the Global Setup expecting that for each (Runtime, MyProperty1, MyProperty2, T) the Global Setup runs just once. So what I did expect was

• Benchmark Settings (Runtime =.NET 8.0, MyProperty1 = 0, MyProperty2 = 0, T = int)
  • A_Global_Setup
  • A_Iteration_Setup_Benchmark1 (Multiple times...)
  • Benchmark1()
  • A_Iteration_Setup_Benchmark2 (Multiple times...)
  • Benchmark2()
• Benchmark Settings (Runtime =.NET 8.0, MyProperty1 = 0, MyProperty2 = 0, T = long)
  • A_Global_Setup
  • A_Iteration_Setup_Benchmark1 (Multiple times...)
  • Benchmark1() (Multiple times...)
  • A_Iteration_Setup_Benchmark2 (Multiple times...)
  • Benchmark2() (Multiple times...) and so on Well since this is not the case I am wondering about 2 things. 1) Is this the expected behavior, if yes, the name global setup seams a bit missleading. 2) Is there a way to enforce my expected behavior so that Global Setup is running not everytime but only once for each setting?

[timcassell]

https://github.com/dotnet/BenchmarkDotNet/issues/2198#issuecomment-1319035590

[swtrse]

@timcassell I almost get a guilty conscience because I always have to follow up and disagree. I have now repeated the test with the Config (DebugInProcessConfig). GlobalSetup is also called several times in this case even if the setting parameters do not change.

[timcassell]

Disagree with what? I mentioned in that comment that it is ran once per benchmark. If you want it to only run once total, you have to make it behave that way manually.

[swtrse]

Ah I get it. I did misunderstood your comment in the other issue.