microsoft / vstest

Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
MIT License
869 stars 316 forks source link

Thread safety is not working properly on .net 6.0 and .net 8.0 for mstest #5011

Closed sampath-ganesan closed 2 months ago

sampath-ganesan commented 2 months ago

Description

I am running mstest test case in multiple thread and when i do that data from one thread to another thread

Steps to reproduce

  1. create mstest + reqnroll project
  2. set Parallelize this is the one i am using ` 2 MethodLevel

    `

  3. and in the Hooks class file i am initializing scenariocontext which is from reqnroll [Binding] public class Binding { private readonly ScenarioContext scenarioContext;

    public Binding(ScenarioContext scenarioContext) { this.scenarioContext = scenarioContext; } }

  4. when i use scenario context in the other hooks like this public class Binding { private readonly ScenarioContext scenarioContext;

    public Binding(ScenarioContext scenarioContext) { this.scenarioContext = scenarioContext; }

    [AfterStep] public AfterStep() { Console.WriteLine(this.scenarioContext.StepContext.StepInfo.Text) } }

another example

private readonly ScenarioContext scenarioContext;
private int randomValue;

public Binding(ScenarioContext scenarioContext)
{
    this.scenarioContext = scenarioContext;
}

[BeforeScenario]
public BeforeScenario()
{
    this.randomValue = Randomizer.GetRandomNumberInRange(0, 100); //I am using custom package in my project to generate random numbers
}

[AfterStep]
public AfterStep()
{
  Console.WriteLine($"Random Number: {this.radomValue}"); 
  Console.WriteLine($"Thread.CurrentThread.ManagedThreadId");
}

when its printing the value both are getting mixed up for example thread Id 14 assign randomValue as 1 thread Id 15 assign randomValue as 85

when its running thread id 14 its taking randomValue as 85 instead of 1

nohwnd commented 2 months ago

This does not look like mstest code, AfterStep or similar attributes are not use by mstest. Which testing framework are you actually using?

I am also not sure what you are showing in your example, the code is most likely running by using Tasks, which won't translate 1:1 to threads.

sampath-ganesan commented 2 months ago

i am using specflow testing framework

sampath-ganesan commented 2 months ago

in this example, where i am setting the random value on each thread and that should be isolated right? but when Parallelize on .runsettings its picking up the last threads value instead of its own thread value

private readonly ScenarioContext scenarioContext; private int randomValue;

public Binding(ScenarioContext scenarioContext) { this.scenarioContext = scenarioContext; }

[BeforeScenario] public BeforeScenario() { this.randomValue = Randomizer.GetRandomNumberInRange(0, 100); //I am using custom package in my project to generate random numbers }

[AfterStep] public AfterStep() { Console.WriteLine($"Random Number: {this.radomValue}"); Console.WriteLine($"Thread.CurrentThread.ManagedThreadId"); }

note: i am running it on 2 threads

nohwnd commented 2 months ago

I am not familiar with the internals of SpecFlow and how they run their setups. But in general the setup can either be run on the same thread. Or it can run in a task, which most likely won't end up on the same thread, and this is perfectly normal, and a desirable behavior of Task.

I cannot tell from your example what you are trying to achieve by checking if you run on the same thread, but in all cases I would expect that the this. reference will refer to the same object in BeforeScenario, and in AfterStep. But that is again a detail of SpecFlow, that this repository does not manage. Please ask specflow for help: https://github.com/SpecFlowOSS/SpecFlow